Skip to content

Unable to complete the operation against any hosts and Host has been marked down or removed on simple query #616

@cezarmoise

Description

@cezarmoise

Driver version: 3.29.5

Code is simple, just create a session are run a query.
Test was run with fips and encription.

with self.cluster.cql_connection_patient(self.target_node) as session:
            query_result = session.execute('SELECT keyspace_name FROM system_schema.keyspaces;')

Error

  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 2074, in _get_all_tables_with_no_compact_storage
    query_result = session.execute('SELECT keyspace_name FROM system_schema.keyspaces;')
  File "cassandra/cluster.py", line 2679, in cassandra.cluster.Session.execute
  File "cassandra/cluster.py", line 5054, in cassandra.cluster.ResponseFuture.result
cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {<Host: 10.4.1.84:9042 eu-west-1>: ConnectionException('Host has been marked down or removed'), <Host: 10.4.1.3:9042 eu-west-1>: ConnectionException('Host has been marked down or removed'), <Host: 10.4.2.116:9042 eu-west-1>: ConnectionException('Host has been marked down or removed'), <Host: 10.4.1.31:9042 eu-west-1>: ConnectionException('Host has been marked down or removed')})

In node logs there is no indication that any node sees another node as down.

Replicated multiple times(example1, example2, example3), but not all the time (example4)


Argus

Scylla version: 2026.1.0~dev-20251208.a213e41250df with build-id e914baa69768b13de7b77c016533dd2edd6df40f

Kernel Version: 5.4.0-1021-aws-fips

Extra information

Installation details

Cluster size: 6 nodes (i4i.4xlarge)

Scylla Nodes used in this run:

- longevity-fips-master-db-node-aec4e38c-1 (3.252.199.184 | 10.4.3.157) (shards: 14)


- longevity-fips-master-db-node-aec4e38c-2 (54.246.41.230 | 10.4.1.31) (shards: 14)


- longevity-fips-master-db-node-aec4e38c-3 (3.253.18.89 | 10.4.3.24) (shards: 14)


- longevity-fips-master-db-node-aec4e38c-4 (34.246.223.89 | 10.4.1.3) (shards: 14)


- longevity-fips-master-db-node-aec4e38c-5 (3.250.35.76 | 10.4.2.116) (shards: 14)


- longevity-fips-master-db-node-aec4e38c-6 (54.154.41.223 | 10.4.1.84) (shards: 14)

OS / Image: ami-0a1dfe766d508f280 (aws: N/A)

Test: longevity-100gb-4h-fips-test
Test id: aec4e38c-f1a6-4104-8ef1-2dab603563d4
Test name: scylla-master/features/FIPS/longevity-100gb-4h-fips-test

Test method: longevity_test.LongevityTest.test_custom_time

Test config file(s):

Logs:

Jenkins job URL

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions