Ports and protocols used by TDP

System

• SSH
  • Port: 22
  • Protocol: SSH
  • External access:
    • All nodes for administrators and deployment scripts
    • Edges nodes for user access to a working pre-configured environment

HDFS

• NameNode
  • HTTPS service The namenode secure HTTP server address and port. It provides access to the HDFS web UI.
    • Port: 9871
    • Protocol: HTTPS
    • Property: dfs.namenode.https-address
    • External access: yes
    • Source
  • RPC service Main RPC port used by client to communicate with HDFS using a binary protocol. The port is embedded in the URI, eg hdfs://nn1.domain.com:8020/.
    • Port: 8020
    • Protocol: IPC
    • Property: dfs.namenode.rpc-address
    • External access: yes
    • Sources:
      • HDFS default configuration
• ZKFC
  • RPC access RPC port used by Zookeeper Failover Controller.
    • Port: 8019
    • Protocol: IPC
    • Property: dfs.ha.zkfc.port
    • External access: no
    • Sources:
      • HDFS default configuration
• DataNode
  • Secure data transfert The datanode server address and port for data transfer. The value depends on the usage of SASL to authenticate data transfer protocol instead of running the DataNode as root, learn more about securing the DataNode.
    • Port: 9866 (SASL based IPC, non-privileged port) or 1004 (privileged port)
    • Protocol: IPC
    • Property: dfs.datanode.address
    • External access: no
    • Sources:
      • Privileged port
      • Non-privilege port
      • HDFS-9427- not default to ephemeral ports
  • HTTPS service The datanode secure HTTP server address and port. It is used to access the status, logs, etc, and file data operations when using WebHDFS or HttpFS. The NameNode UI redirects the user to the DataNode server when browsing files.
    • Port: 9865
    • Protocol: HTTPS
    • Property: dfs.datanode.https.address
    • External access: yes
    • Sources:
      • HDFS default configuration
  • RPC service The DataNode RCP server address and port used for metadata information.
    • Port: 9867
    • Protocol: IPC
    • Property: dfs.datanode.ipc.address
    • External access: (requires Apache Knox)
    • Sources:
      • HDFS default configuration
      • HDFS-9427 - not default to ephemeral ports
• JournalNode
  • RPC server The JournalNode RPC server address and port.
    • Port: 8485
    • Protocol: IPC
    • Property: dfs.journalnode.rpc-address
    • External access: no
    • Sources: HDFS default configuration
  • HTTPS server The address and port the JournalNode HTTPS server listens on. If the port is 0 then the server will start on a free port.
    • Port: 8481
    • Protocol: HTTPS
    • Property: dfs.journalnode.https-address
    • External access: no
    • Sources:
      • HDFS default configuration

YARN

• ResourceManager
  • Ressource tracker This is used by the Node Manager to register/nodeHeartbeat/unregister with the ResourceManager.
    • Port: 8031
    • Protocol: IPC
    • Property: yarn.resourcemanager.resource-tracker.address
    • External access: no
    • Sources:
      • YARN default configuration
  • RPC server The address of the applications manager interface in the RM. It is used to submit jobs. In YARN non HA configuration yarn.resourcemanager.address uses port 8050. In YARN HA configuration, yarn.resourcemanager.address is redundant and instead yarn.resourcemanager.address.{id} is resolved and uses port 8032.
    • Port: 8032
    • Protocol: IPC
    • Property: yarn.resourcemanager.address.{id}
    • External access: yes
    • Sources:
      • YARN default configuration
      • YARN HA port
  • HTTPS server The HTTPS adddress of the RM web UI application. It is used to monitor applications.
    • Port: 8090
    • Protocol: HTTPS
    • Property: yarn.resourcemanager.webapp.https.address
    • External access: yes
    • Sources:
      • YARN default configuration
  • Admin RPC server It is used by administrators and developers.
    • Port: 8033
    • Protocol: RPC
    • Property: yarn.resourcemanager.admin.address
    • External access: yes
    • Sources:
      • YARN default configuration
  • Scheduler It is used by administrators and developers.
    • Port: 8030
    • Protocol: RPC
    • Property: yarn.resourcemanager.scheduler.address
    • External access: no
    • Sources
      • YARN default configuration
• NodeManager
  • Container Manager The address of the container manager in the NodeManager. Access is typically granted to admins, and Dev/Support teams.
    • Port: 0 (default for dynamic port allocation) or 45454 (static port by convention)
    • Protocol: RPC
    • Property: yarn.nodemanager.address
    • External access: yes
    • Sources
      • YARN default configuration
  • Localizer Address where the localizer IPC is. It is responsible for downloading and copying remote resources on the local filesystem.
    • Port: 8040
    • Protocol: RPC
    • Property: yarn.nodemanager.localizer.address
    • External access: no
    • Sources YARN default configuration
  • HTTPS server WebUI server of the NodeManager for administrator and developers.
    • Port: 8044
    • Protocol: HTTPS
    • Property: yarn.nodemanager.webapp.https.address
    • External access: yes
    • Sources
      • YARN default configuration
  • MapReduce ApplicationMaster Ephemeral HTTPS ports are opened by each ApplicationMaster. The tdp-collection default port range is unrestricted but is parameterized in the tdp_var_defaults/hadoop/hadoop.yml inventory file under the yarn.app.mapreduce.am.job.client.port-range property. Ports within this range can be accessible from outside the cluster if the permitted by the network firewall. Note that this only restricts the port range used for mapreduce jobs. In the case of Spark, refer to the Spark Driver port documentation, such as the spark.driver.port property.
    • Port: Random
    • Protocol: HTTP
    • Property: yarn.app.mapreduce.am.job.client.port-range
    • External access: no
• App Timeline Server
  • RPC server This address for the timeline server to start the RPC server. It addresses the storage and retrieval of application’s current and historic information in a generic fashion.
    • Port: 10200
    • Protocol: RPC
    • Property: yarn.timeline-service.address
    • External access: yes
    • Sources
      • YARN default configuration
  • HTTPS server The web UI of the timeline server.
    • Port: 8190
    • Protocol: RPC
    • Property: yarn.timeline-service.webapp.https.address
    • External access: yes
    • Sources
      • YARN default configuration

MapReduce Job History Server

• Job History RPC server Server where client applications submit MapReduce jobs, {FQDN}:{PORT}.
  • Port: 10020
  • Protocol: RPC
  • Property: mapreduce.jobhistory.address
  • External access: yes
  • MapReduce default configuration
• Job History WebUI The MapReduce JobHistory Server Web UI, {FQDN}:{PORT}. It is used by administrators and developers.
  • Port: 19890
  • Protocol: HTTPS
  • Property: mapreduce.jobhistory.webapp.https.address
  • External access: yes
  • Sources
    • MapReduce default configuration
• Shuffle Handler
  • Default port that the ShuffleHandler will run on. ShuffleHandler is a service run at the NodeManager to facilitate transfers of intermediate Map outputs to requesting Reducers.
  • Port: 13562
  • Protocol: RPC
  • Property: mapreduce.shuffle.port
  • External access: no
  • Sources
    • MapReduce default configuration
• RPC admin server The address of the History server admin interface.
  • Port: 10033
  • Protocol: RPC
  • Property: mapreduce.jobhistory.admin.address
  • External access: no
  • Sources:
    • MapReduce default configuration

ZooKeeper

• Client connections Server dedicated to client connections.
  • Port: 2181 (default by convention)
  • Protocol: RPC
  • Property: clientPort in zoo.cfg file
  • External access: yes
  • Sources:
    • ZooKeeper Administrator’s Guide
• Leader server Peers use the former port to connect to other peers, for example, to agree upon the order of updates. More specifically, a ZooKeeper server uses this port to connect followers to the leader.
  • Port: 2888 (default by convention)
  • Protocol: RPC
  • Property: server.x:y:2888:3888 in zoo.cfg file
  • External access: no
  • Sources
    • ZooKeeper Administrator’s Guide
• Leader election connections Server connections used during the leader election phase.
  • Port: 3888 (default by convention)
  • Protocol: RPC
  • Property: server.x:y:2888:3888 in zoo.cfg file
  • External access: no
  • Sources:
    • ZooKeeper Administrator’s Guide

Hive

• Hive Metastore
  Store metadata information to expose data storage into a relational model.
  • Port: 9083
  • Protocol: RPC
  • Property: hive.metastore.uris and hive.metastore.port
  • External access: yes
  • Sources:
    • Hive configuration properties
• Hive Server 2
  The JDBC/ODBC interface to the Hive Metastore.
  • Port: 10000 (in TCP) or 10001 (in HTTP)
  • Protocol: RPC
  • Property: hive.server2.thrift.port
  • External access: yes
  • Source
• Web User Interface (UI)
  Port the the web UI to provides configuration, logging, metrics and active session information.
  • Port: 10002
  • Protocol: HTTPS
  • Property: hive.server2.webui.port
  • External access: yes
  • Sources:
    • Hive configuration properties

Ranger

• Policy Manager Port for Ranger secured admin web UI.
  • Port: 6182
  • Protocol: HTTPS
  • Property: ranger.service.https.port
  • External access: yes
  • Sources:
    • Ranger admin site template

Oozie

• Web UI
  Port for the secured Oozie web UI.
  • Port: 11443
  • Protocol: HTTPS
  • Property: OOZIE_HTTPS_PORT
  • External access: yes
  • Sources
    • Oozie installation guide
• Admin The admin port Oozie server runs. It may be opened externally if job submissions are accepted from outside the cluster.
  • Port: 11001
  • Protocol: HTTPS
  • Property: OOZIE_ADMIN_PORT
  • External access: no
  • Sources:
    • Oozie installation guide

Spark

• Driver port If the spark.driver.port fails, it is incremented by 1 and retried up to spark.port.maxRetries times. spark.blockManager.port must be larger than spark.driver.port + sparkspark.sport.maxRetries. These parameters are identical for Spark2 and Spark3.
  • Port: Random
  • Protocol: HTTPS
  • Properties:
    • spark.driver.port
    • spark.port.maxRetries
    • spark.blockManager.port
  • External access: yes
  • Sources
    • Spark firewall configuration
    • Spark3 configuration docs

Knox

• Gateway The port of Knox main gateway to internal cluster services.
  • Port: 8443
  • Protocol: HTTPS
  • Property: gateway.port
  • External access: yes
  • Source

Additionnal resources

• CDH ports
• CDP ports
• HDFS default configuration
• YARN default configuration
• MapReduce default configuration
• HDFS should not default to ephemeral ports