Whilst I can shed no light on why this has happened, you could try the following:
Make a backup copy of your database file before starting.
Rename the database (which I will call “file” here) from file.dtsparse to file.sparseimage (or file.sparsebundle, depending on which option you used along the way). Open that file from Finder (with DiskImageMounter) - you will be asked for the password. When the file is mounted, you can open it and should find your database inside.
That would be my first step to see whether the file is “valid”; I think in theory you can now pull the database file from the mounted sparse image and drop it wherever you keep your databases; opening it should open the (now unencrypted) database in DT. Next steps would presumably be to copy the content to a new encrypted database (although others may shed more light on options here, and I think once I had checked the database was still healthy I might might wait for input from DT support or others here in the community)
When you have the unencrypted database in front of you, you can show the contents by option-clicking in Finder and selecting “show contents” or similar. Whilst you should never make changes to the database this way, you could verify that your files are contained in it.