Scalable JSON schema and data validation

This work was from 2020 but never publish

The data layer has a supervisory role in checking the data in each dataset, performing type assertions and value checking. Often we will need to check the validity of data; for example, a patient’s temperature, the value should be numeric and perhaps between specific values, and a system should not accept values outside the limits we have set. An extreme value, for example, could be -5 degrees Celsius. How could we avoid such errors? Indeed, one way to deal with the above errors is to check eac value every time, but the above way is inefficient, especially on large data sets, the time it will take to process will be long. It would be ideal and practical if there was a way to validate the data before it even reaches the pipeline . Regardless, it would be ideal if we could use validation in the body of the HTTP response or by the database cursor. Also, there is some uncertainty with the values that can be accepted in each field in our data. However, we provide some documentation explaining what data can be accepted; for example, we need to describe and understand what the data represents and when the data is valid. For example, the JSON Schema is a vocabulary that allows us to define and also perform validation of JSON documents. We could say that JSON schema is a way of describing data that helps us to create a powerful object model

A. JSON Schema

Data schema specifies our expectations about the format and the type of the data indicated and determines what data Identify applicable funding agency here. If none, delete this. is optional or required. It can also help us create more effi- cient representations with compression. Furthermore, it allows systems to evolve independently of each other. figure Schema example schema Data Schemas allow application development without the available data, but as long as the data schema is available, data can be generated for the application development. Data Schemas allow application development without the available data, but as long as the data schema is available, data can be generated to implement the application. Data schemas are critical in data systems and software engineering. Although, the schemas help systems evolve independently of each other and are valuable for integration applications and testing. Ad- ditional, describe the expected keys, value types, and whether specific keys are optional or mandatory. Similarly, it can be used to create more efficient representation models. 1) AVRO Schema: Avro is a data serialization system that uses binary compression , furthermore Avro Shema records de- fined in JSON and include a required name such as user more- over my optional include a namespace such as domain.com. Avro protocols describe in RPC interfaces like schemas they are defined with JSON text likewise a protocol a string name of the protocol (required), namespace an optional string that qual- ifies the name . The name and namespaces qualification rules defined for schema objects apply to protocols as well. Ref- erence [https://avro.apache.org/docs/1.8.2/spec.htmlschemas] Avro records are made up of complex and primitive types. Complex types allow nesting and advanced functionality and could be records, enum, maps, arrays, union or fixed. Figure 3 is an example of Avro schema . Apache Avro provides the flexibility to declare the data schema in various ways , i.e. the definition of the Avro data schema is not unambiguous. One way is to use the namespace as illustrated in figure15. But we could not use namespace in the fields. Further the fields as we have mentioned could be complex type i.e. .... [we will describe each type] Additionally there is nested data schema , in figure 100 is an example with nested 2 levels. All the previous features make it difficult to check the similar Avro schema, but such a tool would be useful and that is why the JSDV tool was implemented.

B. JSON Validation

Data validation is a crucial stage that ensures the informa- tion is correct, and we could associate it with the united test for the software application. The data validation is essential to perform at the Data pipelines, especially for the machine learning pipelines that we could activate a task depending from data flow. At the pre-processing stage, we could apply the data validation or apply it before data desegregation. However, we could use it in both places. The data test could be deterministic or non-deterministic. A data test is deterministic when it verifies the attributes of the data that can be measured without uncertainty. ..... Figure of non deterministic There is a need to provide information integrity and validity, and we have to improve the integration of information assur- ance. We can achieve this by providing a data inspection layer that can provide evidence of data presence while maintaining data security, confidentiality, and data privacy. Moreover, an operation requires proof of data validity under data integrity and proof of any process taken concerning those data. We could adopt techniques for a system that can automatically provide us with additional information, as an example for generating data for verification and validation. A test is non-Deterministic when it measures a quantify with ... We apply Data validation to ensure that the assumptions we have about our data are correct and remain correct. We need data validation because we want to detect bugs -errors in the process step or to directly see the changes in the input from the data. Usually, changes in the input will give significant variation. Additionally, JSON validation is essential and can be a part of the machine learning pipeline. There should be an automatic system to score the changes and also the quality of the data. For example, if we update the data structure, we will immediately have insight into the changes since we could score the similarity of the data to the data schema. Additionally, we could create different validation conditions depending on our data. A typical example is time-series data, where we want all the information in the Test set to be in the future concerning the training data. / section