Upgrade-RHOP-R4.1.1.1 to R4.1.2 build-19 cassandra failed to come up on analyticsdb

Bug #1800547 reported by shajuvk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Invalid
High
alexey-mr
R4.1
Fix Committed
High
alexey-mr

Bug Description

analyticsdb node cassandra service failed to come up after upgrade

After upgrading the cluster from R4.1.1 to R4.1.2, we are removing the entry 'num_tokens: 256' from cassadra.yaml causing the issue.

While we upgrade, do not remove num_token entry.

Workaround:
======

After setting 'num_tokens: 256' in /etc/cassandra/conf/cassandra.yaml
restart service contrail-database

command: service contrail-database restart

log:
 ===

 [main] 2018-10-29 20:16:14,806 StorageService.java:825 - Starting up server gossip
INFO [main] 2018-10-29 20:16:14,904 TokenMetadata.java:479 - Updating topology for /10.0.0.60
INFO [main] 2018-10-29 20:16:14,904 TokenMetadata.java:479 - Updating topology for /10.0.0.60
ERROR [main] 2018-10-29 20:16:14,966 CassandraDaemon.java:708 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot change the number of tokens from 256 to 1
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:989) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:682) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:379) [apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) [apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [apache-cassandra-3.11.2.jar:3.11.2]
INFO [StorageServiceShutdownHook] 2018-10-29 20:16:14,977 HintsService.java:220 - Paused hints dispatch
INFO [StorageServiceShutdownHook] 2018-10-29 20:16:14,977 Gossiper.java:1540 - Announcing shutdown
INFO [StorageServiceShutdownHook] 2018-10-29 20:16:14,978 StorageService.java:2292 - Node /10.0.0.60 state jump to shutdown
INFO [CompactionExecutor:2] 2018-10-29 20:16:16,805 PerSSTableIndexWriter.java:275 - Scheduling index flush to /var/lib/cassandra/data/ContrailAnalyticsCql/messagetablev3-a3255230db0d11e8812e718c1d8dee9d/mc-5-big-SI_messagetablev3_column10_idx.db

Tags: upgrade rhosp
shajuvk (shajuvk)
information type: Proprietary → Public
summary: - Upgrade-RHOP-R4.1.1.1 to R4.1.2 build-19 cassandra failed to come up
+ Upgrade-RHOP-R4.1.1.1 to R4.1.2 build-19 cassandra failed to come up on
+ analyticsdb
shajuvk (shajuvk)
Changed in juniperopenstack:
assignee: nobody → alexey-mr (alexey-morlang)
description: updated
description: updated
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/47374
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
alexey-mr (alexey-morlang) wrote :

not applicable to master

Changed in juniperopenstack:
status: New → Invalid
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/47374
Committed: http://github.com/Juniper/puppet-contrail/commit/957cddf2048584cd9f53fa39e208c876fd70cf86
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 957cddf2048584cd9f53fa39e208c876fd70cf86
Author: alexey-mr <email address hidden>
Date: Tue Oct 30 22:47:15 2018 +0300

Fixed reding of num_tokens for cassandra

Change-Id: I17df1c37f8dbb5240fe4d897e687be866030658a
Closes-Bug: #1800547

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/47418
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/47418
Committed: http://github.com/Juniper/puppet-contrail/commit/037f86e39f050b146c3586be634d873892ff79fa
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 037f86e39f050b146c3586be634d873892ff79fa
Author: alexey-mr <email address hidden>
Date: Thu Nov 1 15:32:59 2018 +0300

Convert num_tokens facter to number from string

Partial-Bug: #1800547

Change-Id: I8c08195dd72854cbc5889655bdb236a2216ac9b3

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.