MAAS

Blank DNS entry won’t delete

Bug #2007192 reported by Gregory Orange
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Triaged
Medium
Unassigned
MAAS 3.4.x

Bug Description

We have found a DNS entry in our MAAS 3.2.6 setup which causes a problem: It has no name value, so from MAAS the whole subdomain resolves to an unwanted IP, while from outside of MAAS it is already configured to resolve correctly. On the region controller:

maas:~$ head -12 /etc/bind/maas/zone.nimbus.pawsey.org.au
; Zone file modified: 2022-10-27 19:11:26.026174.
$TTL 30
@ IN SOA nimbus.pawsey.org.au. nobody.example.com. (
              0000986051 ; serial
              600 ; Refresh
              1800 ; Retry
              604800 ; Expire
              30 ; NXTTL
              )

@ 30 IN NS pawsey.org.au.
 30 IN A 146.118.52.233

Notice the space starting the last line - by my reading that is why nimbus.pawsey.org.au resolves to that IP from MAAS. Externally it resolves correctly from upstream DNS to a different IP. At https:/.../MAAS/r/domain/1 (Name: nimbus.pawsey.org.au) I see what I think is the erroneous entry, screenshot at https://discourse.maas.io/t/blank-dns-entry-wont-delete/6436 of me trying to delete it.

maas SESSION dnsresources read | grep 146.118.52.233 finds the IP, but it is in an ‘ubuntu’ dnsresource record in another domain pawsey.org.au which also shows an entry in the web UI. That entry holds a lot of IPs, a few dozen named ‘ubuntu’. I think this is not relevant but am not sure.

$ maas SESSION dnsresource read 1899 | jq '.fqdn'
"ubuntu.pawsey.org.au"
$ maas SESSION dnsresource read 1899 | wc -l
117074
$ maas SESSION dnsresource read 1899 | jq '.ip_addresses | .[].ip' | wc -l
1511

I am left wondering if this snuck in quite some time ago (perhaps in an enlistment gone wrong?) in a previous version of MAAS, and in the current version that particular type of record can’t exist, and can’t be deleted. Do I need to dive into the database and delete something manually to fix this?

May I have some advice on this please?

Questions from @cgrabowski:
1. you have a domain (DNS zone with regards to BIND) pawsey.org.au, and then you have a delegated zone of nimbus.pawsey.org.au, are either of these the default MAAS domain? And then in one of these , you have an ubuntu.pawsey.org.au A record, but then the erroneous empty label record with the same IP?
2. Additionally, what is that IP tied to? Is it a machine within MAAS? If so, what is the status (i.e is it deployed?)?

Answers to the above:
1. In short, yes what you said is accurate. I’m not sure about it being a ‘label’ record, but in maas domain pawsey.org.au (htt.../MAAS/r/domain/0) there is a Resource record like so:

ubuntu A (default) 146.118.52.233

Any attempt to change or delete it yields sequence item 0: expected str instance, NoneType found. I can’t find any evidence of a machine with that IP address - nothing from arp or nmap either.

2. The IP isn't tied to anything that we know about. It does not respond, and nmap -Pn gives `All 1000 scanned ports on 146.118.52.233 are filtered`.

Gregory Orange (gregoryo2017)
summary: - MAAS: Blank DNS entry won’t delete
+ Blank DNS entry won’t delete
Revision history for this message
Igor Brovtsin (igor-brovtsin) wrote :

Hi Gregory. Could you please check if `regiond.log` has any errors reported when you try to delete these entries? If there are, please add them to the bug as well.

Changed in maas:
status: New → Incomplete
Revision history for this message
Gregory Orange (gregoryo2017) wrote :

Apologies for the delay, I messed up my email filters. Thanks to @cgrabowski for prompting me on Discourse.

Yes, error message attached, first 2 and last 2 lines here. Nothing else meaningful appears to have happened immediately before or after these lines in the log.

2023-03-03 12:21:54 maasserver.websockets.protocol: [critical] Error on request (10) domain.delete_address_record: DNSResource matching query does not exist.
 Traceback (most recent call last):
...
     raise self.model.DoesNotExist(
 maasserver.models.dnsresource.DoesNotExist: DNSResource matching query does not exist.

Gregory Orange (gregoryo2017)
Changed in maas:
status: Incomplete → New
Revision history for this message
Anton Troyanov (troyanov) wrote :

Hi Gregory,

Not sure if you still have this issue, but here are my thoughts:

I've checked API (both utilised by CLI and UI) and there is no way to create DNS entry with an empty name (or name containing whitespaces)

But as for database schema, empty/whitespace string is a valid value (but so far I failed to find any code that would create such entries):

maasdb=# \d maasserver_dnsresource
                                       Table "public.maasserver_dnsresource"
   Column | Type | Collation | Nullable | Default
-------------+--------------------------+-----------+----------+---------------------------------
 created | timestamp with time zone | | not null |
 updated | timestamp with time zone | | not null |
 name | character varying(191) | | |
 domain_id | integer | | not null |
 address_ttl | integer | | |

I believe my best advice is to manually delete bogus entry from the database table `maasserver_dnsresource` as well as do some cleanups on the referenced tables.

But before deletion, can you please check `created` and `updated` values for that record?

Changed in maas:
status: New → Incomplete
Revision history for this message
Gregory Orange (gregoryo2017) wrote :

Hello Anton,
Thank you for the response. Firstly, I would be content to update the database directly but it would be easier if I knew how to follow some of the foreign keys. That is, maasserver_dnsresource table has no identifying information in it, and oddly enough it only has 41 rows as compared to over 400 machines controlled by MAAS.

Previously I was looking at the name 'ubuntu', so just looking at that, the entry is
 1899 | 2020-03-23 14:34:54.656713+08 | 2020-03-23 14:34:54.656713+08 | ubuntu | 0 |
(Oh hang on, ID 1899 matches, but I am still not sure how accurate that is.)

However, I am now seeing new behaviour. From both rackd and regiond servers:

$ dig nimbus.pawsey.org.au +short
146.118.53.173
146.118.52.233

and sure enough:

$ head -13 /etc/bind/maas/zone.nimbus.pawsey.org.au | tail -3
@ 30 IN NS pawsey.org.au.
 30 IN A 146.118.53.173
 30 IN A 146.118.52.233

146.118.53.173 was marked as Discovered on a baremetal node in Ready state which I have been working with today. I have just deployed it, and it has been autoassigned a different IP in that subnet, but the DNS entry remains.
146.118.52.233 is marked as Discovered on a baremetal node which has for many months been deployed as a MAAS KVM Pod host, although it too has a different autoassigned IP in that subnet.

What should I do now?

Adam Collard (adam-collard)
Changed in maas:
milestone: none → 3.4.0
Gregory Orange (gregoryo2017)
Changed in maas:
status: Incomplete → New
Bill Wear (billwear)
Changed in maas:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Gregory Orange (gregoryo2017) wrote :

May I have some advice on this?

Alberto Donato (ack)
Changed in maas:
milestone: 3.4.0 → 3.4.x
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.