org.apache.cassandra.transport.ProtocolException: Invalid or unsupported protocol version: 4

When I wanted to upgrade my Cassandra from 2.1.3 to 3.4 in Windows 7 64 bit, I have faced following error when starting cqlsh console.

E:\apache-cassandra-3.4\bin>cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1': ConnectionException(u'Did not get expected SupportedMessage response; instead, got: <ErrorMessage code=0000 [Server error] message="io.netty.handler.codec.DecoderException: org.apache.cassandra.transport.ProtocolException: Invalid or unsupported protocol version: 4">',)})

This was happening even when I downgrade my cqlversion as follows.

E:\apache-cassandra-3.4\bin>cqlsh --cqlversion=3.4.0
Connection error: ('Unable to connect to any servers', {'127.0.0.1': ConnectionException(u'Did not get expected SupportedMessage response; instead, got: <ErrorMessage code=0000 [Server error] message="io.netty.handler.codec.DecoderException: org.apache.cassandra.transport.ProtocolException: Invalid or unsupported protocol version: 4">',)})

E:\apache-cassandra-3.4\bin>python -V
Python 2.7.6
E:\apache-cassandra-3.4\bin>cqlsh --version
cqlsh 5.0.1

This is due to protocol version (4) unsupported by cql.
Edit <cassandra_home>\bin\cqlsh.py to make following inclusions so that it accepts --protocolversion argument along with cqlsh command. It's advisable to take backup of cqlsh.py file before editing it.

I have given edited lines below. Add only the lines that are marked with + (plus sign). Other lines are given for reference of places to make change. 

.
.
parser.add_option('--cqlversion', default=DEFAULT_CQLVER,
                  help='Specify a particular CQL version (default: %default).'
                       ' Examples: "3.0.3", "3.1.0"')
+ parser.add_option("--protocolversion", default=DEFAULT_PROTOCOL_VERSION, help='Specify protocol version (default: %default).')
parser.add_option("-e", "--execute", help='Execute the statement and quit.')

.
.
    try:
        port = int(port)
    except ValueError:
        parser.error('%r is not a valid port number.' % port)

+    if options.protocolversion:

+       try:
+            options.protocolversion = int(options.protocolversion)
+        except ValueError:
+            options.protocolversion=DEFAULT_PROTOCOL_VERSION

    return options, hostname, port

.
.
                      connect_timeout=options.connect_timeout,
                      encoding=options.encoding,
 +                    protocol_version=options.protocolversion)
    except KeyboardInterrupt:
        sys.exit('Connection aborted.')

Once the changes are done, save it and restart Cassandra server. Now execute cqlsh with following command.

E:\apache-cassandra-3.4\bin>cqlsh --protocolversion=3

Still you may face any issue like following.

Connection error: ('Unable to connect to any servers', {'127.0.0.1': ProtocolError("cql_version '3.4.0' is not supported by remote (w/ native protocol). Support
ed versions: [u'3.2.0']",)})

In such case, execute cqlsh command as follows and the issue would be resolved now.

E:\apache-cassandra-3.4\bin>cqlsh --protocolversion=3 --cqlversion=3.2.0

Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.3 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
WARNING: pyreadline dependency missing.  Install to enable tab completion.
cqlsh>

JVM Heap Memory Architecture

Heap divided into different sections shown below :

YoungGen :

 It is place where lived for short period and divided in two parts:

Eden Space : 

When object created using new keyword memory allocated on this space.

Survivor Space : 

This is the pool which contains objects which have survived after java garbage collection from Eden space.

Tenured Generation : 

This memory pool contains objects which survived after multiple garbage collection means object which survived after garbage collection from Survivor space.

OldGen : 

This pool is basically contain tenured and virtual (reserved) space and will be holding those objects which survived after garbage collection from YoungGen space

Permanent Generation : 

This memory pool as name also says contain permanent class metadata and descriptors information so PermGen space always reserved for classes and those that is tied to the classes for example static members.

 Java8 Update: PermGen is replaced with Metaspace which is very similar. Main difference is that Metaspace re-sizes dynamically i.e., It can expand at runtime. Java Metaspace space: unbounded (default)
 

Code Cache (Virtual or reserved) : 

If you are using HotSpot Java VM this includes code cache area that containing memory which will be used for compilation and storage of native code.

Java.Lang.OutOfMemoryError: PermGen Space

One of the least understood areas by Java Developers is garbage collection. Java Developers feel JVM takes care of garbage collection and they need not worry about memory allocation, deallocation etc. But as the applications grows complex, so does the garbage collection and once it is complex, the performance do get a hit. So it will benefit the Java Developers to understand how garbage collection works and how to fix the ‘Out of Memory’ issues in java. There are 2 quite common ‘Out of Memory’ issues. The 1st one is ‘Heap Size’ and the 2nd one is ‘PermGen Space’.

Permanent Generation and ClassLoaders

Java objects are instantiations of the Java classes. Every time a new java object is created, JVM creates an internal representation of that object and stores it in the heap. If the class is accessed for the first time, then it has to be loaded by the JVM. Class loading is the process of locating the corresponding class file, seeking the file on the disk, loading the file and parsing the structure. It is the ClassLoaders responsibility to ensure proper loading of the classes.Each and every class in the java program needs to be loaded by the same ClassLoader. ClassLoaders are the instances of java.lang.ClassLoader class. For now, ClassLoader loads the java classes in Perm Space.

JVM also creates an internal representation of the java classes and those are stored in the permanent generation. During garbage collection, both java objects and classes are viewed as objects and are garbage collected in the same way. Initially both the java objects and classes are stored in the heap space.
As a performance optimization the permanent generation was created and classes were put into it.Classes are part of our JVM implementation and we should not fill up the Java heap with our data structures. Permanent Generation is allocated outside the heap size. The permanent Generation contains the following class information:

•     Methods of a class.
•     Names of the classes.
•     Constants pool information.
•     Object arrays and type arrays associated with a class.
•     Internal objects used by JVM.
•     Information used for optimization by the compilers.

Now that we understood what Permanent Generation is, let us see what causes the memory issue in this region.

PermGen Space

‘Java.Lang.OutOfMemoryError: PermGen Space’ occurs when JVM needs to load the definition of a new class and there is no enough space in PermGen. The default PermGen Space allocated is 64 MB for server mode and 32 MB for client mode. There could be 2 reasons why PermGen Space issue occurs.

The 1st reason could be your application or your server has too many classes and the existing PermGen Space is not able to accommodate all the classes.

-XX:MaxPermSize=XXXM

If the issue is due to insufficient PermGen Space due to large number of classes, then you can increase the PermGen space by adding the –XX:MaxPermSize=XXm parameter. This will increase the space available for storing the classes and should -XX:MaxPermSize=256m

-XX:+CMSClassUnloadingEnabled

This parameter indicates whether class unloading enabled when using CMS GC. By default this is set to false and so to enable this you need explicitly set the following option in java options.

-XX:+CMSClassUnloadingEnabled

If you enable CMSClassUnloadingEnabled the GC will sweep PermGen, too, and remove classes which are no longer used.This option will work only when UseConcMarkSweepGC is also enabled using the below option.

-XX:+UseConcMarkSweepGC

-XX:+CMSPermGenSweepingEnabled

This parameter indicates whether sweeping of perm gen is enabled. By default this parameter is disabled and so will need to explicitly set this for fine tuning the PermGen issues. This option is removed in Java 6 and so you will need to use -XX:+CMSClassUnloadingEnabled if you are using Java 6 or above. So the options added to resolve the PermGen Space memory issues will look like

-XX:MaxPermSize=128m -XX:+UseConcMarkSweepGC XX:+CMSClassUnloadingEnabled

Memory leaks

And the 2nd reason could be memory leak. How the class definitions that are loaded could can become unused.

Normally in Java, classes are forever. So once the classes are loaded, they stay in memory even if that application is stopped on the server. Dynamic class generation libraries like cglib use lot of PermGen Space since they create a lot of classes dynamically. Heavy use of Proxy classes, which are created synthetically during runtime. It’s easy to create new Proxy classes when a single class definition could be reused for multiple instances.

Spring and Hibernate often makes proxies of certain classes. Such proxy classes are loaded by a classloader. The generated class definitions are never discarded causing the permanent heap space to fill up fast.

For PermGen space issues, you will need identify the cause of leak and fix it. Increasing the PermGen space will not help, it will only delay the issue, since at some point the PermGen space will still be filled up.