January, 2009
Hadoop and Linux kernel 2.6.27 – epoll limits
Yesterday we faced a strange problem. A newly set up Hadoop cluster got unstable after a few minutes. Logs reported a lot of exceptions like: java.io.IOException: Too many open files at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method) at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:68) at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:52) at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18) at sun.nio.ch.Util.getTemporarySelector(Util.java:123) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:92) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:281) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102) at java.lang.Thread.run(Thread.java:619) or DataXceiver java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:298) at [...]