I recently ran across the “TFS TF400983: There is an error in XML document” error on our Team Foundation Server 2013.4 on-premises installation. I received an alert from our production TFS environment following a failed backup the previous evening. The TF400983 error appeared in the TFS logs, and an error “TF53010: The following error has occurred in a Team Foundation component or extension” was present in the Application event log. TF53010 is a generic error, so the devil is in the details as they say:
TF53010: The following error has occurred in a Team Foundation component or extension:
Date (UTC):
Machine: MY_TFS_SERVER
Application Domain: TfsJobAgent.exe
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v4.0.30319
Service Host:
Process Details:
Process Name: TFSJobAgent
Process Id: 1344
Thread Id: 2496
Account name: {TFS_SERVICE_ACCCOUNT}
Detailed Message: TFS database backup job failed with error: There is an error in XML document (1191, 66).System.Xml.XmlException: There is an unclosed literal string. Line 1191, position 66.
at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.ParseAttributeValueSlow(Int32 curPos, Char quoteChar, NodeData attr)
at System.Xml.XmlTextReaderImpl.ParseAttributes()
at System.Xml.XmlTextReaderImpl.ParseElement()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read4_BackupSetDatabase(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read5_TfsBackupSet(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read6_BackupSets(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read7_BackupSets()
The backup log file can be found here: "\SERVER_BACKUPServerBackupsTFS-BackupsFull_9859122861445405443.log"
Exception Message: There is an error in XML document (1191, 66). (type InvalidOperationException)
Exception Stack Trace: at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
at Microsoft.TeamFoundation.Admin.BackupSets.Load(String folder)
at Microsoft.TeamFoundation.Admin.Jobs.DatabaseBackupJob.RunFullBackupJob(TeamFoundationRequestContext requestContext, BackupSettings settings, ITFLogger logger)
at Microsoft.TeamFoundation.Admin.Jobs.DatabaseBackupJob.RunBackupJob(TeamFoundationRequestContext requestContext, BackupJobType requestedJobType, String& resultMessage)
Inner Exception Details:
Exception Message: There is an unclosed literal string. Line 1191, position 66. (type XmlException)
Exception Stack Trace: at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.ParseAttributeValueSlow(Int32 curPos, Char quoteChar, NodeData attr)
at System.Xml.XmlTextReaderImpl.ParseAttributes()
at System.Xml.XmlTextReaderImpl.ParseElement()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read4_BackupSetDatabase(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read5_TfsBackupSet(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read6_BackupSets(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read7_BackupSets()
Per the recommended guidance, we use the “Scheduled Backups” feature located within the TFS Admin Console. Upon investigation, I saw the following in our log file:
[8/13/2015 5:07:39 PM] [Info] Requested Backup Job: Full [8/13/2015 5:07:39 PM] [Info] ------------------------------------------------------------------- [8/13/2015 5:07:39 PM] [Info] Jobs Requested: Full. Completed:None. To Run:Full. [8/13/2015 5:07:39 PM] [Info] RunFullBackupJob [8/13/2015 5:07:39 PM] [Error] Exception Message: There is an error in XML document (1191, 66). (type InvalidOperationException) Exception Stack Trace: at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events) at Microsoft.TeamFoundation.Admin.BackupSets.Load(String folder) at Microsoft.TeamFoundation.Admin.Jobs.DatabaseBackupJob.RunFullBackupJob(TeamFoundationRequestContext requestContext, BackupSettings settings, ITFLogger logger) at Microsoft.TeamFoundation.Admin.Jobs.DatabaseBackupJob.RunBackupJob(TeamFoundationRequestContext requestContext, BackupJobType requestedJobType, String& resultMessage) Inner Exception Details: Exception Message: There is an unclosed literal string. Line 1191, position 66. (type XmlException) Exception Stack Trace: at System.Xml.XmlTextReaderImpl.Throw(String res, String arg) at System.Xml.XmlTextReaderImpl.ParseAttributeValueSlow(Int32 curPos, Char quoteChar, NodeData attr) at System.Xml.XmlTextReaderImpl.ParseAttributes() at System.Xml.XmlTextReaderImpl.ParseElement() at System.Xml.XmlTextReaderImpl.ParseElementContent() at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read4_BackupSetDatabase(Boolean isNullable, Boolean checkType) at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read5_TfsBackupSet(Boolean isNullable, Boolean checkType) at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read6_BackupSets(Boolean isNullable, Boolean checkType) at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderBackupSets.Read7_BackupSets() [8/13/2015 5:07:39 PM] [Info] Backup Job Finished. Completed: None. Result = Failed
The last line of the stack trace refers to “Read7_BackupSets”, which made me think of the BackupSets.xml file, located int he destination database backup location. After some additional investigation, I found the BackupSets.xml file was corrupted. In fact, it appears that the file had been partially written, and was clearly invalid XML.
During my investigation, I also found a blog post from Rich Hewlett, describing a similar issue. In Rich’s case, the BackupSets.xml file seemed complete – in mine, half of the file was missing.
As Rich pointed out, the solution was easy – rename/delete the file, and then re-run the full backup. After renaming the file, the backup succeeded.
