This is an example installation on DigitalOcean. For sure there are many more paths how-to install OpenEndpoints on DigitalOcean.

1. Sign up to DigitalOcean or be invited to an existing account

2. On the left hand navigation go to “API” and “Generate New Token” with a name such as your name. Record the token somewhere for later.

3. Create a Project

   1. Top-left of the navigation all projects are listed

   2. Click on “New Project”

   3. Enter the name “Endpoints”

   4. For “What’s it for” answer “Operational / Developer tooling”
4. Upload the Endpoints software as a Docker image to DigitalOcean Docker Repository

   1. Go to “Container Registry” on the left hand side navigation

   2. Use a name which is unique to your company or product; this is globally unique across all of DigitalOcean. We use “endpoints” in this example.

   3. On your local computer (e.g. Mac): Install "doctl", see https://docs.digitalocean.com/reference/doctl/how-to/install/

5. Create a database:

   1. Click on “Databases” in left navigation

   2. Click "create"

   3. Choose “PostgreSQL”
6. Create the application:

   1. On the left navigation click on “App Platform”

   2. Create a new app

   3. Select "source provider" of “DigitalOcean Container Registry”

After you've published your application, you probably want to test your endpoint :-)

Calculate Hash

1. Navigate to Calculate Hash in the main navigation.

2. Select your endpoint and the environment.

3. In case your endpoint has "include-in-hash" parameters, you will be prompted to enter values for those parameters. See: .

4. Press button Calculate Hash and copy the calculated hash value.

Build the Link

1. Navigate to Home in the main navigation.

2. Copy Live URL or Preview URL - depending on what environment you would like to use. The URL should look like this: https://endpoints.openendpoints.com/foo/{endpoint}?{parameter}

3. Replace {endpoint} with the name of your endpoint

Test the Link

1. Copy the link into your browser.

2. Navigate to Request-Log in the main navigation. You will find your request, including additional information in case an error has occurred.

The entire configuration of an application that is built with OpenEndpoints consists of files in a directory - we refer to this as the Application Directory.

An example directory structure is available on Github (public repository):

For each configuration file or directory, additional explanations are available in the subsections of this page.

Note that is not a working example application. The aim of the example directory is to present the expected syntax and directory structure.

Application loaded from Git

The application directory resides in your own Git repository. There are 2 :

• Multi Application Mode (=default): The Service Portal provides a simple user interface to "publish" the latest version from Git into the software. An integrity check with meaningful error messages is carried out automatically. Most configuration errors are detected in this way.

• Single Application Mode: You create a new Docker image derived from the standard Docker image, which includes the application directory. You need to build a new Docker image each time your configuration has changed.

The multi-application mode also offers the possibility of simple staging - see .

Expected XML sometimes less strict than XSD

The software is a little less strict than what is written in the XSD: Don’t be surprised if you find working examples where the order of elements is slightly different from what is described in the XSD.

Additional files silently ignored

Any additional files which are present in the application directory, but not required by the software, are silently ignored. For example, if you create an additional directory "project files" that will be ignored by the software without raising an error.

Minimum configuration requirement

Many configuration files are only required if the corresponding features are used.

In any case, these few files must be present for a working configuration:

• endpoints.xml having at least 1 endpoint

• security.xml having at least 1 secret key

• at least one transformer, which as a matter of facts also requires a data-source

The optional xml-from-application directory contains custom xml files that may be used as a data-source. There are no restrictions on how these files should be structured.

The idea behind locally stored content is to avoid unnecessary data load if content doesn't change often.

The optional directory data-source-xslt contains custom xslt files.

Each transformer references to a data-source, which can be transformed to any output having some custom data-source-xslt file.

See: Data Source Transformation

Transformation Input

The data-source will generate a file with a root tag <transformation-input> comprising of all content generated by the data-sources.

The optional directory data-source-post-processing contains custom XSLT files.

See: .

Transformation Input

The root tag of the input xml (which is automatically provided by the software) is <data-source-post-processing-input>. Within that root tag OpenEndpoints will insert the data converted from the data-source-definition or - if applicable - the output from a previous data-source-post-processing step.

This optional directory contains custom XSLT for .

Parameter Transformation Input

If an endpoint uses the parameter transformation, OpenEndpoints automatically generates the temporary XML parameter-transformation-input.xml, which is the input for the transformation. This is (unless debug is enabled) only stored in memory, and never written to disk. Depending on how data are submitted to the endpoint, the input XML might look slightly different.

The mandatory directory data-sources contains data-source definitions. A data-source-definition contains references to one or many data-sources, which you would like to load data from and convert to output using a transformer.

See: .

OpenEndpoints can interact with third party REST or SOAP APIs, either using such API as a data-source, or to execute a TASK.

The request body of such request can be either inline content (=directly written into endpoints.xml or into the data-source definition), or the body is generated by XSLT transformation.

This optional directory contains custom xslt to generate such request bodies.

The transformation input (=the XML generated by the software, which the XSLT is applied to) does not support arbitrary data-sources, but is limited to parameter values:

The values of the parameters inside that generated xml are different depending on the context:

You can embed custom fonts into your generated PDF.

In order for custom fonts to be embedded in a PDF, 2 steps must be set:

1. Upload the TTF files into the fonts directory. Only TTF fonts are supported.

2. Declare those fonts in the file apache-fop-config.xsd.

The optionalstatic directory can be used to store any non-XML local content, for example images and prefabricated PDF. You may use subdirectories.

Images in PDF

On inserting images into PDF, the static directory serves as the local root path to the XSLT processor. So you can either use an absolute path (URL) or a relative (local) path that refers to this directory.

The mandatorytransformers directory contains one or many transformers defining a Data Source Transformation.

The only mandatory attribute is the data-source. Omitting all other tags will output the original data-source (xml) without any transformation.

This type of syntax specifies that an endpoint "step-1" is executed and on success another endpoint "step-2" will be called.

The result of the original request (i.e. all parameters) is used as the request to the endpoints being forwarded to:

<endpoint name="step-1">
    <success>
        <forward-to-endpoint endpoint-name="step-2"/>
    </success>
</endpoint>

All parameter values are forwarded to the new endpoint.

System parameters such as user agent, client IP address and file uploads are all available at the endpoint forwarded to. They are inherited to the forwarded endpoint.

It’s possible to chain the execution of any number of endpoints in this manner (e.g. endpoint e1 forwards to e2 which itself forwards to e3). A circular chain of such references is not allowed as the processing of such a chain would never end.

The “redirect” from one endpoint to another happens within the Endpoints software; no redirect is actually sent to the user’s browser.

Request Log Behaviour

Only one “request log” gets written, despite a chain of multiple endpoints being processed. Only the first “parameter transformation input/output” is saved with that “request log” entry, despite each endpoint in the chain potentially having its own parameter transformation.

This lists the most recent object keys (filenames) out of the bucket specified in the aws-s3-configuration.xml file (see example-customer for example format.).

“Most recent” means the keys of the objects with the most recent last modified timestamp.

The command looks like:

<aws-s3-keys limit="100">
    <folder>foo/bar</folder> <!-- optional -->
    <match-tag name="foo">bar</match-tag>
    <match-tag name="abc">def</match-tag>
</aws-s3-keys>

and the results look like:

<aws-s3-keys>
    <object key="folder/xyz.xml"/>
    ...
</aws-s3-keys>

Literal XML

The data-source command <literal-xml> lets you define xml output directly and "literally" in the data-source definition file.

The root tag <literal-xml> is not included in the data-source xml output. In the example above, the generated xml will be:

This data-source-type can be perfectly used in combination with parameter placeholders. For example, you can use something like this:

If ${foo} equals "hello world", the data-source output will be:

Note that the contents of must be elements, simply placing text straight under the <literal.xml> element will not work.

Application Introspection

This content-source produces as its output a description of the entire application directory structure (=your configuration).

The generated content has a root-tag <application-introspection> and returns

• <directory name="x"> for any directory

• <file name="x"/> for all XML files. The content of the XML file is included as a child of this tag, except the directory xml-from-application. (Use the <xml-from-application> data source, not <application-introspection> to load content from such files.)

On-Demand Incrementing Number

<OpenEndpoints/> can generate unique auto-increment values and provide them as a data-source. Read for more details.

Parameter Transformation is an option for the advanced processing of inputs from a request. It allows to

1. have different names for parameters submitted and parameters declared in endpoints.xml

2. produce parameter values different from submitted values

3. validate input data and create custom error messages, implementing custom validation rules

4. process XML body as input (content type "application/xml")

To use parameter-transformation add a tag <parameter-transformation> to the endpoint-definition in the endpoint-folder. The xslt (mandatory) is from the directory "parameter-transformation-xslt" under application. You may use subdirectories to organize XSLT in this directory.

Optionally zero or many data sources may be added. Syntax for adding data sources is the same as for .

Default behaviour without parameter transformation

In the absence of "parameter transformation" the default behaviour for processing data inputs is:

• Data can be sent as GET or POST parameters only. It is not possible to supply request type "application/xml" as an input.

• The name of the parameter sent must be existing as a parameter-name in endpoints.xml.

• Sending parameters not existing in endpoints.xml will cause an error.

• For every parameter existing in endpoints.xml a value must be submitted, unless a default-value exists. Otherwise this will cause an error.

Alternative behaviour with parameter transformation

1. If the request is a GET/POST with parameters then all parameters are taken and <parameter name="x" value="y"/> elements are created - no matter if declared in <endpoints.xml> or not. If the request is a POST with an XML body then the XML is taken as is.

2. Any optionally specified data sources are executed. Any GET and POST parameters may be accessed with the ${x} syntax (see the data source descriptions for where parameters may be used). Any parameter which is referenced but not supplied with the request is left empty (an error is not produced) as the point of the parameter transformation is to determine errors. An error being produced by a missing parameter would then not allow the parameter transformation to produce a custom <error> output.

3. Input from [1] and [2] are placed into an XML called "parameter-transformation-input.xml". See for details.

Inputs from request and data from optionally added data sources are automatically placed into a temporary XML called <parameter-transformation-input>.

The custom XSLT ("parameter-transformation-xslt") will be applied to this XML.

OpenEndpoints will automatically insert additional useful tags:

Tags added to <input-from-request>

The parameter-transformation-xslt will use parameter-transformation-input.xml as an input and will generate parameter-transformation-output.xml.

The required output scheme is:

• Each parameter existing in endpoints.xml must be present in the output, except if the parameter has a default-value (which will be applied if the parameter were missing in the output).

• Output of a parameter not existing in endpoints.xml will raise an error.

Any task can be made conditional, that means the task will only be executed if some parameter value matches a condition.

<task class="..." if="${foo}" equals="true">
    ...
<task>

The current set of operators supported are:

• if="..." equals="..."

• if="..." notequals="..."

• if="..." isempty="true"

• if="..." hasmultiple="true"

• if="..." gt="..."

• if="..." ge="..."

• if="..." lt="..."

• if="..." le="..."

Note the syntax of the if condition: Either side can use parameter placeholder.

If the parameter has a value like foo||bar i.e. created as a result of a request such as ?param=foo&param=bar, then the equals=".." will check if any of the values match, and notequals=".." will check that none of the values match the value.

For the gt, ge, lt, le operators the comparison values will be treated as numbers (decimal). If either side are empty or not parseable as a number, the comparison is false.

The right hand side of isempty and hasmultiple can be true or false.

The application can be deployed in two different ways.

In both scenarios

• Docker is used to deploy the application.

• A PostgreSQL database is required. No schema is needed, the application creates that itself the first time it runs.

• We recommend using a cloud with managed Docker services, e.g. ECS on AWS, or Kubernetes.

Multiple Application

This is the default option. You deploy the standard Docker image, you don't need to build your own.

The Service Portal is part of the installation:

• Applications are published from Git using the Service Portal.

• Application files are not stored inside the Docker image.

Single Application

This is a special option.

With this deployment mode the application directory is stored directly inside the Docker image. You build your own Docker image with your own Application files being part of that container.

Hence, only a single application will be available with this installation.

OpenEndpoints is software written in Java. The code is open source and licensed under Apache License with Commons Clause - see https://openendpoints.io/license.

For the deployment we recommend using Docker. A public Docker container is available at:

public.ecr.aws/x1t6d0t7/endpoints-he

Using Saxon-PE

By default the public Docker container will install the free XSLT processor Saxon-HE. This is sufficient for most purposes.

In order to deploy OpenEndpoints with the commercial version Saxon-PE you are required to buy a license for Saxon-PE (Professional Edition) and create a Docker container using that edition. The license isn't expensive and it's absolutely worth the money.

Here are the steps you need to take:

1. Purchase a Saxon-PE license at . You will get two files: the JAR file containing the Saxon-PE code, and also a Saxon-PE license file.

2. Install Java 21 and Maven on your computer if you have not already done so.

3. Check out the OpenEndpoints Git repo to your computer if you have not already done so.

Get Access to an AWS Account

Firstly you need to either create an AWS account, or you need to have access to an AWS account.

Decide on Domain

Decide on what internet domain the new Endpoints deployment should be reachable at. For example https://endpoints.myservice.com/.

This can either be a domain such as myservice.com or a subdomain such as endpoints.myservice.com. In either case, you must own the main domain you wish to host on (myservice.com in this example). Purchasing this domain is out-of-scope of this document.

Apply for an HTTPS Certificate via AWS

AWS offers free HTTPS certificates for use with their own services.

1. In the AWS Management Console (web interface), navigate to the Certificate Manager product.

2. Click “Request a Certificate”

3. On the form that is displayed, select “Request a Public Certificate” as this will be a publicly visible service.

4. Type in the domain name of the certificate; this is the full domain name, from the example above this would be

Create VPC and Subnets

A VPC is a part of the network managed by AWS. If this AWS account will do nothing more than serve Endpoints, you can use the default VPC. If it is shared with other services, for security we recommended creating a separate VPC for the Endpoints installation. If you decide to use a default VPC, you can skip this section. If you decide to use a separate VPC, follow the following steps to create it:

1. In the AWS Management Console (web interface), navigate to the VPC product.

2. Create a new VPC, with the appropriate IP address range, such as 10.100.0.0/16. (Ignore the options about IPv6.) You can use any internal IP address range you like, as long as it doesn't conflict with any other internal IP addresses that the VPC may need to have access to (e.g. if Endpoints needs to be configured to access any databases or services internal to your company). If in doubt, talk to the department which manages IP addresses in your company. If the VPC does not need to access any other internal services, there is no reason not to proceed with the example given here.

3. Create two Subnets, with IP address schemes like 10.100.10.0/26 and 10.100.10.64/26. Allow “Specify the Availability Zones” explicitly, choose different zones for each Subnet, otherwise they all get created in the same Availability Zones. This allows services to be split across multiple Availability Zones, meaning if one AWS data center encounters issues, then the application will still be available.

Create Security Groups

It is a good idea to create all security group in advance. Security groups can also be created when databases and other resources are created, however then they have useless names such as “rds-launch-wizard-2”.

On the AWS Management Console (web interface), navigate to the Security Group feature. The following table specifies which security groups to create. Each have a name and one or more inbound rules. For outbound rules, allow the default which is to be able to send requests on any port to any location.

Create "Bastion" VM

For various tasks, it is useful to have a VM to connect to via SSH. This is "behind the firewall" and will allow you to access resources such as the database which are not public.

1. In the AWS Management Console (web interface), navigate to EC2 product.

2. If you do not already have an SSH public key then go to the left navigation under "Key Pairs" and create a new Key Pair. If you already have an SSH public key, e.g. created on your computer, you do not need to do this step, you can upload it later.

3. Navigate to the "Instances" section of the left-hand navigation.

It takes quite a while before a user can log in to a newly created EC2 instance, for example 5 minutes. You might see the error "Permission denied (publickey)" during this time.

To connect to it, use the ssh username ubuntu together with the key you created or uploaded earlier.

Create PostgreSQL Database

Endpoints uses the database to store various things such as which Endpoints "applications" have been installed.

1. In the AWS Management Console (web interface), navigate to RDS product. This is the AWS managed database product.

2. Click “Create a new database”.

3. Name it something like “Endpoints”.

4. We currently support PostgreSQL 14 (although other versions will probably work).

Create the Task Definition

A Task Definition is a blueprint of how AWS will install and run the Endpoints software.

1. In the AWS Management Console (web interface), navigate to the ECS product. ECS is the service AWS offers to manage Docker installations.

2. Create a new Task Definition

3. Select type "Fargate". This means that AWS itself will automatically allocate compute resources, as opposed to having to do it manually.

4. Name the Task Definition with a name like "Endpoints"

Create an ECS Cluster

A "Cluster" is a set of compute resources, managed by AWS, where Endpoints will run.

1. In the AWS Management Console (web interface), navigate to the ECS product

2. Create the ECS cluster (not Kubernetes Cluster)

3. Select “Networking only” type.

4. Don’t select creation of a new VPC

Create a Load Balancer (ELB)

The Load Balancer is responsible for taking the user's HTTPS requests, and forwarding them on to the Endpoints software running on a managed ECS cluster created above.

1. In the AWS Management Console (web interface), navigate to the ECS product

2. Go to the “Load Balancers” section.

3. Click “Create Load Balancer”.

4. Select the default “Application Load Balancer” from the two options.

Connect the Domain to the Load Balancer

This is necessary so that when someone navigates to your domain, their requests are sent to the AWS Load Balancer created above, and thus the request can be served by Endpoints.

1. In the AWS Management Console (web interface), go to the EC2 product.

2. In the Load Balancer section, click on the Load Balancer created above. You will see it has a DNS name such as Endpoints-HTTPS-66055465.eu-central-1.elb.amazonaws.com (A Record).

3. In the tool where you administer the domain, create a CNAME DNS record, from the domain or subdomain chosen for this installation, to the domain name you can read off in the last step.

Add Services to the Cluster

This step takes the Task Definition you have created earlier (which is a blueprint for running the software) and installs it on the Cluster created earlier (a set of compute resources).

1. In the AWS Management Console (web interface), go to the ECS product.

2. Navigate to ECS “Clusters” (not Kubernetes Clusters)

3. Select the newly created “Cluster”.

4. Create an service called “Endpoints”

Create Monitoring Alarms

It is possible, but not necessary, to use CloudWatch to create monitoring alerts for the health of various components such as the database. To create an alarm:

1. In the AWS Management Console (web interface), go to the CloudWatch product.

2. Click “Create Alarm”.

3. Set up the alarm, as described below

4. On the last screen, select that the action should be to send an email to the appropriate person.

Perform the above steps for each of the following alarms:

• CPU: ECS -> CPUUtilization, > 70 (percentage) for 1 period (= 5 minutes)

• Memory: ECS -> MemoryUtilization > 70 (percentage) for 1 period (= 5 minutes)

• Up: ApplicationELB > Per AppELB, per TG Metrics -> UnHealthyHostCount, > 0 for 1 period (= 5 minutes)

Use Application

The application is now available under your URL e.g. https://endpoints.myservice.com/

If the application is configured in multi-application mode then the Service Portal is available under https://endpoints.myservice.com/service-portal with username admin/admin.

Troubleshooting

• Go to the ECS Cluster, go to the Service, click on the “Events” tab to see what’s going on, e.g. health check is failing

• Go to CloudWatch, Log Groups, see that there is a new log group which has been created, go into the log file and see if there are any errors.

Description

This software takes XML from multiple sources (e.g. URLs, files, or any Java class you write which can produce XML). You configure what sources you want, and the software combines the XML from those sources into one XML file in memory. The XML is then optionally transformed using XSLT. The resulting document is then optionally converted into PDF, XLS or JSON. The resulting file is then either downloaded to the browser, or a sent as an email, or you can write a Java class which performs any action on the file, or any combination of the above.

A PostgreSQL database is required prior to installing OpenEndpoints.

Database schema creation is not necessary; the software does that on startup.

The following data is saved in it:

• Request log: This saves metadata for each call, but not the data explicitly transmitted when the call is made. An exception is the debug mode, in which, as an exception, the data transferred when the call is made can also be saved.

• Incremental IDs: If the feature is used to provide incremental IDs, the last ID used is saved in the database.

The term ”application” is used to describe a piece of configuration. OpenEndpoints can serve multiple applications simultaneously. You can create different projects with different application directories and run them in parallel.

An application is configured with a directory of files - mainly XML files. The structure of the directory is explained in detail here: .

OpenEndpoints pulls the application directory from a Git. When publishing, the last version is pulled from the Git and your configuration is updated.

Add or Edit Application

Click Add New Application

HTML web forms allow to send files along with the form data:

OpenEndpoints automatically understands the correct enctype to separate uploaded content from form <input> data:

A simple web form to send data to OpenEndpoints could look like this:

What shall happen after submit?

The form calls your endpoint, which might for example

• use an to forward an email to your back office, and also send a custom message to the email-address submitted with the form.

An application comprises of multiple endpoints. Each endpoint has a name which is used in the URL. Which endpoint is requested is specified when the user calls the application.

OpenEndpoints installed in "Multi Application Mode" (see ) comes with a service portal for administration:

https://[your-base-url]/service-portal

Note that in "Single Application Mode" your application is already packed with the Docker container.

Default Login

A default account for the admin is automatically created during installation:

A distinction must be made between two technically different topics:

• The web form containing multiple controls having the same name attribute.

• The web form containing a single control, but which allows multiple values to be selected.

By default, OpenEndpoints does not save any data that was transferred with a request.

Sometimes, however, it can make sense to have data transfer tracking available for error analysis, for example. This is what the debug mode and verbose is for.

Working Principle

With the parameter transformation, OpenEndpoints generates a parameter-transformation-input.xml and - after successful transformation - a parameter-transformation-output.xml. These are only stored in memory and not persisted to disk, by default.

When Debug Mode is activated, these two files are saved in the request log and can be downloaded from the Service Portal. (If the transformation fails, the output.xml is not generated).

The mandatory file endpoints.xml under the application directory comprises of one or many endpoint folders, each of which contains of

• one or many endpoint definitions

• zero or many

The optional file service-portal-endpoint-menu-items.xml enables the creation of additional, user-defined pages in the service portal.

Pages of type form and content can be optionally organized under a menu-folder. The maximum supported depth of menu items is one.

Form

This type of syntax specifies that the transformation named in the attribute is performed, and its contents are returned to the client in the HTTP response. For example, the transformation might produce HTML to be displayed in the browser, or a PDF to be downloaded.

Syntax

You can optionally add an attribute "download-filename".

If the download-filename="foo.pdf" attribute is present, then the header in the HTTP response is set indicating that the file should be downloaded as opposed to displayed in the browser window. Parameter values like ${foo}

The current configuration of an application is loaded from a GIT to OpenEndpoints (with the "Publisher" in the service portal).

To enable a simple form of "staging", 2 different versions of the GIT repository can be activated at the same time. There are 2 different "environments" for this: PREVIEW and LIVE.

Publisher

The "Publisher" in the service portal lets you

In endpoints.xml, <endpoint> has two sub-sections, <success> and <error>. What happens in the case of success or error depends on the tags being present. Details are described in the following subsections.

If the tag (<success> or <error>) is missing, or present and empty, this means the server returns an empty 200 OK in the success case and 400 error in the case of failure. This can be useful if the request should simply perform some tasks e.g. send emails.

Even on a static website there is always content that changes regularly and can be loaded from a database or a web service, for example. OpenEndpoints is perfect for converting external content into HTML, which can be seamlessly integrated into your own website.

HTML Sniplets

Build an . Do not use the download-filename attribute in this case.

The generated HTML is inserted directly into the <div>. Therefore, the HTML (generated with XSLT) should not contain a <head> or <body> tag, but actually only the content that is to be inserted!

Each request to OpenEndpoints requires a mandatory hash parameter:

The hash is a SHA-256 hex calculated from several input values. On each request the server calculates the expected value of the user-supplied hash parameter. If the supplied value does not match the expected value the request will be denied.

The hash may be supplied as uppercase or lowercase in the request.

The mandatory file security.xml under the application directory comprises of one or many secret keys, which are used to calculate hash keys. See: Authentication

The beauty of <OpenEndpoints/> is its potential to produce custom content on demand. Practically, that means: Data submitted from a webform (or any other method of submitting data) can directly influence

• what content is loaded from which data sources

• what exactly to transformation of a content shall look like.

Endpoint parameters come from the following sources:

1. Parameters submitted from the originating request to the endpoint including, but not limited to, ?x=y GET parameters

2. The result of a parameter transformation (for example, extracting parameters from arbitrary XML sent as a POST)

3. An "intermediate value" generated by one task and consumed by future items (for example, results from an HTTP request which was sent during the processing)

This data source type will output all available parameter values and make it available for data-source transformation.

The generated content looks like this:

A special type of parameter value is an uploaded file. For example if you are submitting data from a webform which has <input type="file" name="my-upload"/>, then the presentation of input depends on the type of uploaded content.

If the uploaded content can be parsed as XML, this xml will be available in the data-source:

If the uploaded content can not be parsed as XML, the output is:

Intermediate parameters are not regular endpoint parameters, i.e. they are not defined as a <parameter> in endpoints.xml and their value does not come from the original request. Intermediate outputs are generated from tasks that execute during the processing of the request. On forwarding an endpoint to another endpoint those parameters will be made available in the parameters-data-source as well:

By default the root-tag of the generated output is <parameters>. Use the optional tag attribute to generate any different root-tag:

The optional file email-sending-configuration.xml under the application directory is required if emails are to be sent from OpenEndpoints.

OpenEndpoints supports two options for sending emails:

• Sending via SMTP

• Sending via MX address

For further details about how to send emails see:

Basic Principle

<OpenEndpoints/> transforms content from CMS, REST-APIs, databases and static files into other content structure and other content types. The technical solution is XML Transformation.

This requires

1. XML as a data-source

This type of syntax specifies that an HTTP request is performed and the result of this request is streamed back to the client as the response of the call to Endpoints.

Syntax

You can optionally add an attribute "download-filename".

If the download-filename="foo" attribute is present, then the header in the HTTP response is set indicating that the file should be downloaded as opposed to displayed in the browser window. Parameter Values like ${foo} are replaced.

Basic Syntax

The data-source command <xml-from-database> fetches rows from a database and transforms rows and columns into XML.

Currently only MySQL and PostgreSQL are supported. Other databases would require other client JARs that are not provided in the current version of the software.

Alternative option using the environment-variable to connect to your local endpoints database - in this example (sql!) fetching request details from the request-log:

You can fetch content from any REST API and use it as a data-source in <OnestopEndpoints/>.

The data-source command to fetch xml or json data from any URL is <xml-from-url>. JSON or HTML returned to this command will automatically converted into XML.

For JSON to XML conversion:

• Any characters which would be illegal in XML (for example element name starting with a digit) replaced by _xxxx_ containing their hex unicode character code.

A request contains any number of GET or POST parameters.

The values supplied by the user for these parameters are used in various places. For example, if you are fetching XML from a URL, parts of the whole of the URL may be the value of a parameter. The syntax for this is ${param}. The braces are, in contrast to many other systems, mandatory.

Note that for security or technical reasons replacement is not always possible. See the specific sections for the various configuration elements where parameter replacement occurs.

Different results may be required for the same endpoint based on certain criteria. It is possible to define any number of success elements such as:

The conditions are considered in the order they’re written in the file, so put more general “catch-all” items at the bottom and more specific “if...” items at the top.

Parameters may also be used in the "equals" or "notequals" attribute. You could for example create a condition like

Only if=".." equals=".." and if=".." notequals=".." are available.

If the parameter has a value like foo||bar i.e. created as a result of a request such as ?param=foo&param=bar then the equals=".." will check if

This feature can be used to download e.g. Word or Excel files.

Only e.g. DOCX and XLSX etc. are supported; old-style DOC and XLS files are not supported.

The body of the document may contain parameter references such as ${foo}, these are expanded to the parameters available through the endpoint processing.

The endpoint application directory may contain an ooxml-responses directory and within that any referenced files must be present, for example foo.docx in the example above.

This type of syntax specifies that a redirect of the request should be performed. The body of the <redirect-to> specifies where. Variables, if present, are replaced, in the body.

For example you can redirect a successful request to a "thank.you.html" url.

Parameters like ${foo} in the body are replaced.

If you use variables, we recommend to use this optional element to prevent a malicious request redirecting somewhere wrong:

If no such tag is present, redirect to any URL is allowed. If one or more are present, the URL being redirected to must start with the prefix of one of them; otherwise this is an error.

This type of syntax specifies that the response content is fetched from the named file within the static directory. Subdirectories such as filename="sub-dir/file.pdf" are supported.

Variables are not allowed in the filename attribute. This is to prevent accidentally making more files accessible (if the client guesses the right filename) than intended. But you can use to deliver different files depending on different parameter values.

You can optionally add an attribute "download-filename".

If the download-filename="foo.pdf" attribute is present, then the header in the HTTP response is set indicating that the file should be downloaded as opposed to displayed in the browser window. Parameter Values like ${foo} are replaced.

In contrast to content loaded from the internet, XML already existing within the application does not require to be loaded each time. This makes it a good choice for large files. It is possible to use XML files with more than 100,000 lines of content without causing any performance issues. In general, the limiting factor is the performance of the XSLT transformation rather than the size of the data source.

You can place such files under the the xml-from-application directory within the application. You can load the content of this file into the data-source:

You can use placeholders to fill in endpoint parameter values into the file attribute:

If the file can not be found, an error will be raised. To avoid that, add an attribute ignore-if-not-found="true". The data-source in this case will look like if the content was not requested at all.

Transformer file

In the transformers directory under the application, there are zero or more files, each describing a transformation. Subdirectories are not supported.

The root element of each transformer file is <transformer> .

The root <transformer> element has with a mandatory

In general <error> supports the same tags as <success>. But there are limitations.

No forward-to-endpoint

Placing <forward-to-endpoint> inside <error> is not supported, as this would require “auto increment” numbers to be assigned even in the case of <error>

Sometimes it may be required to insert a unique incremental id into a generated content.

Depending on the specific business use case, the incremental id may be unique “perpetual”, or it may be required to re-start the counter every month or every year.

This reads a particular object (file) from AWS S3. It is assumed that this object contains XML data. The command looks like:

<aws-s3-object key="folder/foo.xml"/>

and the results look like:

<aws-s3-object key="folder/foo.xml">
    ... file contents (XML) ...
</aws-s3-object>

There is the possibility of adding instructions to output the input/output of the transformation to AWS S3.

<transformer data-source="spheres-fo">
  <write-input-to-aws-s3>
    <folder>foo/bar</folder> <!-- optional -->
    <tag name="foo">bar</tag> 
    <tag name="abc">def</tag> 
  </write-input-to-aws-s3>
  ...
  <write-output-to-aws-s3> 
    ...
  </write-output-to-aws-s3>
</transformer>

This creates objects in the S3 bucket which have the tags as specified, the correct Content-Type, and in addition a tag called "environment" which is either "preview" or "live".

A task is like a ”secondary action” assigned to an endpoint. The primary action is the <success> and <error> action described in Types of Endpoints. The "secondary" action may contain zero or many tasks, that will be executed once the primary action has been completed successfully.

With a task you can

• send a request to any API =>

• send emails with attachments =>

Each <task> is a set of commands embedded into an endpoint-definition in the endpoints.xml:

Parallel or Subsequent Execution of Tasks

Offer-Ready uses multiple cores, if available. If not specified otherwise, tasks are executed in parallel and will be finished in an arbitrary order. In order to determine a distinctive order of tasks, read .

Intermediate Values

If a task requires the output from a second task as an output, you may use .

OpenEndpoints supports any kind of HTTP request to other systems. You can call and fetch data from

• REST Api

• SOAP interface

• simple URL that returns content

Examples:

• fetch data from any CRM or ERP system (as long as it offers a REST or SOAP API that can be called from the internet)

• upload generated files to a CRM or an archive

• fetch the next available invoice-number from your accounting system, generate an invoice and send it as an email

• validate the existence of an address by calling an external validation service

Basic Syntax

This task performs an HTTP request, checks the response is a 2xx OK, and ignores the response body.

Redirects are not followed.

The attribute ignore-if-error="true" may be present on the <task> element to indicate that if an error occurs (e.g. server not found, non-2xx response, etc.) this error is ignored. By default, the error aborts the processing of the endpoint.

Request Body

💡The beauty of <OpenEndpoints/> shows in the solution of the optional request body, which can be json or xml. There are several different options how-to build the content for the request-body.

XML-Request Body with Inline Contents

The request body is expressed as xml within the <xml-body> tag. Endpoint parameters are expanded.

Uploaded content encoded in base64 can be filled into any tag of the request body. This requires 2 actions:

• Add attribute upload-files="true" to <xml-from-url>

• Add to any element of your request body attributes upload-field-name="foo" encoding="base64"

The uploaded content will expand into that xml element.

It is also possible to send generated content within a request body:

• Add attribute expand-transformations="true" to <xml-from-url>

• Add to any element of your request body attributes xslt-transformation="foo" encoding="base64"

iAdding that attribute to the element indicates that the transformation with that name should be executed (for example, generate a PDF file), and the contents of the resulting file should be placed in this tag. The encoding is always base64, no other encodings are supported.

XML-Request Body from Transformation

The request body is generated by XSLT. This leaves maximum flexibility to build different content of the request body depending on endpoint parameter values!

Note that this is a transformation within a transformation. The XSLT takes a <parameters> as its input document; This XSLT does not have access to the results of any other data sources. The reason is, that data sources cannot use data produced by another data source.

• The XSLT file is taken from the http-xslt directory.

• The transformation-input to apply that XSLT has <parameters> as its root tag.

The optional attribute upload-files="true" and expand-transformations="true" may be present as above.

JSON-Request Body with Inline Contents

The request body is expressed as json within the <json-body> tag. Endpoint parameters are expanded.

Endpoint parameters are expanded within the string context of JSON, that is to say that no concern about escaping is necessary.

Options for expanding base 64 content from file upload or generated content is not available for JSON.

JSON-Request Body from Transformation

The request body is generated by XSLT. That requires that the result of the transformation is valid JSON.

Note that this is a transformation within a transformation. The XSLT takes a <parameters> as its input document; This XSLT does not have access to the results of any other data sources. The reason is, that data sources cannot use data produced by another data source.

• The XSLT file is taken from the http-xslt directory.

• The transformation-input to apply that XSLT has <parameters> as its root tag.

Options for expanding base 64 content from file upload or generated content is not available for JSON.

Endpoint parameters can be used as placeholders in your data source. On executing the data source definition any parameter placeholder will be replaced by the respective value of the parameter.

You can only use parameters defined in the endpoints.xml file of your configuration.

You can not use

• intermediate parameters (from a task)

• content submitted from a file-upload (or - more generally - any additional content submitted from a multi-part message)

Basic Syntax

If my parameter is called "foo" ...

then I can use as a placeholder:

Placeholders in the data-source definition file

Placeholders can be used inside the data-source.xml file.

For example, you can select the specific piece of content loaded from CMS by evaluating a submitted parameter value. Or you could load specific database rows selected for a specific parameter value.

For security or technological reason this does not work in every case. For details please refer to the specific sections:

Inline Placeholders directly in the content source

On loading content from any of these data source types placeholders will be automatically replaced by its parameter values:

• xml-from-application

• xml-from-url

For example, you can use ${foo} as a placeholder directly in your CMS. On loading data from your CMS, actual values will replace the placeholders.

Some email programs may have issues with long links. Links to endpoints (containing all parameters) may get long, so this can become a problem.

The “Short Link To Endpoint” feature allows shorter links to endpoints (including all parameters) to be created. This is analogous to Forwarding Between Endpoints, with the exception that rather than the destination endpoint getting executed immediately, a link is created to the processing of that endpoint.

The task creates a short-link in the database with a random code. The resulting full link, including the code and also including the base URL of the current installation of Endpoints.

The short link looks like this: [base-url]/shortlink/RANDOMCODE.

The generated link is written to an output intermediate variable. The concept of intermediate valriables is described here: Intermediate Values.

The shortlink will be auto-deleted in the database after the time specified in expires-in-minutes. For example, if you put expires-in-minutes="1440" then the link will be available for 1 day. After that time the link will not work any longer.

Use a syntax like the following to create a short link to an endpoint in the variable ${foo}. (You can choose any other variable name, of course).

The variable ${foo} can then be used as an input-intermediate-value in a subsequent task.

For example, you can send an email containing ${foo} in the email-body. The xslt (to create the email-body) would look like this:

Offer-Ready is built around the concept of dynamically generated content. But of course parts of the content required to build your endpoints may not require being dynamically generated, because they already exist as some sort of “static” content.

Static content such as images or files may be stored in the application’s static directory.

Image Paths in PDF

Image paths within an XSL-FO file may have an absolute path (to some URI), or a relative path.

• Absolute paths will be "executed" by the PDF reader. Make sure that the path is accessible from your client's device.

• Relative paths will use the application’s static directory as a root directory. Images will be embedded in the generated PDF.

Offer-Ready uses multiple cores, if available. If not specified otherwise, tasks are executed in parallel and will be finished in an arbitrary order.

Paralell execution of primary action and tasks

Note that - as a default - the primary task (data transformation) and all tasks are executed in parallel.

Therefore if one HTTP request depends on another previous one having completed first, that will not work without declaring this dependency.

You may optionally assign an id attribute to the task element:

<task class="endpoints.task.HttpRequestTask" id="foo">
  ...
</task>

You may insert an element <after task-id="..."/> into any task that needs to be executed after foo…

<task class="endpoints.task.HttpRequestTask">
    <after task-id="foo"/>
    ...
</task>

Note that on using Intermediate Values the software will automatically determine the order of execution such that intermediate outputs are created before they are required as inputs. Intermediate values and “after” elements can be used in parallel.

Depending on what fonts readers have available on their local computer, fonts used in your PDF will be seen as you intended it to be seen, or the computer may substitute fonts. Substitution can result in significant differences between your intended output and what the reader will see.

Embedding fonts prevents font substitution. This ensures that text is displayed and printed in its original font.

True Type Fonts

Offer-Ready supports embedding of True Type fonts, which have a “ttf” file extension.

Create a directory “fonts” in the root directory of your configuration. Within this “fonts” directory, there is

• an overall fonts configuration file called “apache-fop-config.xml”.

• A TTF file for each font refered to in the apache-fop-config.xml. For example: “HelveticaNeue-Light.ttf”.

The “apache-fop-config.xml” declares the ttf fonts in the fop processor:

How-to use your embedded fonts in XSL-FO

On developing your XSL-FO template, use the (embedded) font-triplet name as if it were available on your local computer. Technically your XSL-FO file may contain any font-name with any font-style and font-weight. The behaviour of a common PDF reader will be:

1. If a font-name/font-style/font-weight triplet used in the PDF is available as an embedded font-triplet, than this will be used.

2. If a font-name/font-style/font-weight triplet used in the PDF is not available as an embedded font-triplet, but on the local computer, than the font from the local computer will be used.

3. If both previous options do not apply, the PDF reader will substitute that font by some other font.

Beware of these “traps”:

• If your PDF for example does not show a text in bold letters, although your intention was to use bold letters, than you maybe have defined the font-triplet for “weight=normal”, but not also for “weight=bold”. You need to declare all required combinations of name/style/weight.

• If your local computer has a font “Helvetica Neue”, and your font-triplet name is “HelveticaNeue” (without a blank), than you must use “HelveticaNeue” in your XSL-FO. A WYSIWYG editor (like Altova Stylevision) likely will offer local fonts only in a dropdown.

• The preview of your PDF in a local development environment may look different than the PDF generated by Offer-Ready. The fonts used by the local development environment will use fonts on your computer, and not fonts checked in in the above structure.

Third Party License Constraints

A data source is a list of zero or many commands which fetch content from different content sources and produce XML. The resulting xml output is the input ("transformation-input") for XSLT to generate output documents.

Sometimes it makes sense to apply one or many intermediate steps to modify the loaded content before it becomes a "transformation-input". Possible reasons for this include

• You might want to reuse the same XSLT to generate an output document for different content sources, but these content-sources do not produce the exactly same structure of input. An intermediate transformation step can be used to "normalize" the input among different content sources.

• A complex transformation might be implement more elegant by splitting it into several subsequent steps.

A data-source-post-processinig.xlt can apply xml-transformations within the data-source object.

How does it work?

In the data-source-post-processing-xslt directory under the application, there are zero or more files, each describing a post-processing transformation step. Each file is a XSLT which expects input-data with a root tag <data-source-post-processing-input> and which shall produce any output xml with a root-tag <data-source-post-processing-output>.

Content loaded from source A:

Expected output:

You can add zero or many data-source-post-processing.xslt to each content-source of your data-source object. For each content-source post-processing will be executed separately. Multiple steps for the same content-source will be executed subsequently in the order of post-processing-xslt files.

In addition you can apply the same logic for the entire data-source object. In this case all content-sources are loaded as a first stept, and post-processing applies for the collection of all content-sources.

To send an email from OpenEndpoints you need to

1. Configure your email server

2. Create an email task

Configure Email Server

To send email from your application the file "email-sending-configuration.xml" must be present under application.

The file has the root element <email-sending-configuration> and have the following sets of sub-elements.

• If no username and password are set, TLS will not be used

• extra headers are written into every email sent via SMTP, for example authorization headers for a commercial email sending service

Any of the fields (apart from the header names) may use ${foo} parameters.

Alternative Set-Up:

An alternative option is to configure an MX address for the DNS lookup.

Create Email Task

The task <task class="endpoints.task.EmailTask"> sends an email. It has the following sub-elements configuring it:

• <from> is mandatory (variables are expanded)

• <to> is mandatory (variables are expanded). There may be multiple <to> elements. Each <to> sends a separate email, to just this recipient. Per <to>, only one recipient address is allowed

• <subject> is mandatory (variables are expanded)

• <body-transformation name="a-transformation"/> is mandatory, and can appear multiple times. This references a transformation (see below). All the different results are placed into a ”multipart/alternative” email part. It would be normal for one referenced transformation to produce HTML and the other plain text.

• <attachment-static filename="path/foo.pdf"> takes the foo.pdf file out of the static directory and includes it as an attachment in the email. Variables are not allowed in the filename attribute.

• <attachment-transformation name="a-transformation" filename="invoice-${invoice-number}.pdf"/>. For each of the elements, the transformation is executed, and the resulting bytes are attached as a file to the sent email. The name of the file is specified in the filename attribute, variables are expanded.

• <attachment-ooxml-parameter-expansion source="foo.docx" filename="invoice-${invoice-number}.pdf"/> will read in the file “foo.docx” from the “ooxml-responses” directory under the Endpoint's configuration and replace any ${foo} variables in the document's body, and deliver it. Only DOCX is supported; DOC is not supported. The name of the file is specified in the filename attribute, parameters like ${foo} are expanded.

• <attachments-from-request-file-uploads/>. This includes as attachments all file uploads that have been uploaded to this request. Any attachment may (optionally) have attributes such as if="${foo}" equals="bar".

Embedding Images

If the body has a content type like text/html; charset=utf-8 then it may include tags such as <img src="cid:foo/bar.jpg">. The tag is most commonly an <img> but can be any tag.

The system then searches in the static directory for any file with that path. The file is included with the image, as a “related” multi-part part, meaning the file is available to the HTML document when its rendered in the email client.

Saxon XSLT Processor

By default the Saxon-HE product from Saxonica is used, which is open-source and requires no license fees.

Basically, the software also supports the use of commercial versions of Saxon - see (Link Removed). When Saxon-PE is available, the following additional functions for XSLT will be available in OpenEndpoints:

• <xsl:value-of select="uuid:randomUUID()" xmlns:uuid="java:java.util.UUID"/>

• <xsl:value-of select="math:random()" xmlns:math="java:java.lang.Math"/> - Generate random number between 0 and 1, e.g. 0.37575608763635215.

• <xsl:value-of select="base64:encode('foo')" xmlns:base64="java:com.offerready.xslt.xsltfunction.Base64"/>

• <xsl:value-of select="base64:decode('Zm9v')" xmlns:base64="java:com.offerready.xslt.xsltfunction.Base64"/> - assumption is that the encoded text is UTF-8 text.

• <xsl:value-of select="digest:sha256Hex('foo')" xmlns:digest="java:org.apache.commons.codec.digest.DigestUtils"/>

• <xsl:value-of select="reCaptchaV3:check('server side key', 'token-from-request')" xmlns:reCaptchaV3="java:com.offerready.xslt.xsltfunction.ReCaptchaV3Client"/> yields a number from 0.0 to 1.0, or -1.0 in the case a communication error has occurred (see log for more details of the error)

XSLT Processor Unaware Of XSD

In most cases, the XML that you are transforming will have a schema (xsd) that can be used when developing the XSLT. The XML standard also provides for reference to the xsd directly from the XML.

Our implementation of the XSLT processor does not expect any reference to an XSD, and ignores this reference if it were present.

Global XSLT Parameters

The global section of XSLT can contain parameters which are intended for the transfer of values for controlling the XSLT.

The root element of a transformer (see: ) can have elements like <placeholder-value placeholder-name="x" value="y"/> which will be passed to the XSLT processing as <xsl:param>.

Note that it is not supported to extract an in the value attribute - ${foo} etc. will not work.

Data-Source Transformation Input XML

In the process of developing the data-source-transformation XSLT, it might be useful to get the transformation input xml the XSLT will be applied on.

In order to get the transformation input xml, simply omit the xslt (and all other options) from the transformer to get the raw transformation-input-xml.

Originally XSLT specifically was designed to support a 2-step process of

1. Transformation: In the first step an XML document transforms into another XML document containing information about how to display the document: what font to use, the size of a page, etc. This markup is called Formatting Objects. Note that the resulting XML document not only contains formatting information, but is also stores all of the document's data within itself.

2. Formatting: Some software (called “FO-Processor”) transforms the result of the first step (transformation) into the intended output format. For example, Apache™ FOP (Formatting Objects Processor) is an output independent formatter, which can generate PDF.

OpenEndpoints uses Apache™ FOP to generate PDF.

The data-source transformer refers to a XSLT that creates XSL-FO (=an xml containing content + formatting options).

In the example above the 2-step process is:

1. [xslt-which-creates-xsl-fo] transforms [data-source] into the xsl-fo-output = an xml containing content + formatting options.

2. The option <convert-output-xsl-fo-to-pdf/> triggers the function to send xsl-fo-output to Apache FOP, which will generate PDF.

The Apache FOP integration with OpenEndpoints includes easy to use options to

• into the generated PDF

• Embedding Fonts such as Google Fonts into your generated PDF

Advantages of XSL-FO

XSL-FO is a very mature standard for page composition, which was designed for paged media. It is capable of comprehensive layout functionality, which makes it possible to create error-free but also beautiful layouts. For example: Pagination controls to avoid “widows” and “orphans”, the support of multiple columns, indexing, etc. It is the perfect technology to produce “content-driven” design.

For more advantages and disadvantages of XSL-FO see:

In addition to parameters there are Intermediate Values . These are like parameters, but they do not come from the user’s request, instead they come from other tasks.

For example, imagine a CRM which requires 2 separate calls to

1. fetch the next available id to insert a new customerinsert a new customer, with that id (from irst request) as a mandatory parameter

2. insert a new customer, with that id (from irst request) as a mandatory parameter

In this case the first <task> will fetch the id, and make it available as an input to the second task. This is an “intermediate value”.