Wednesday, September 27, 2017

Transfering PDF files via the VFS transport with WSO2 ESB

This post will explain how one can transfer PDF files through VFS transport within WSO2 ESB.

In this example, I will be providing the configuration which is tested on WSO2 ESB 4.8.1.


In order for you to get the scenario to work, first you will need to enable the VFS sender and listener through the following configuration in the axis2.xml. The below lines will be commented out by default and all you need to do to enable the VFS transport is uncomment the following two entries.

<transportReceiver name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportListener"/>

<transportSender name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportSender"/>


Next, to enable PDF file transferring within ESB, you will have to enable the message relay feature. For this, we need to add the appropriate message builder and formatter to the axis2.xml file.

<messageFormatters>

        <messageFormatter contentType="application/pdf" class="org.wso2.carbon.relay.ExpandingMessageFormatter"/>
        :
        :
</messageFormatters>

<messageBuilders>
         <messageBuilder contentType="application/pdf" class="org.wso2.carbon.relay.BinaryRelayBuilder"/>
          :
          :
</messageBuilders>



Once the above changes have been done, create a Proxy Service as shown below.

   <proxy name="PdfProxy" transports="vfs" startOnLoad="true">
      <target>
         <inSequence>
            <log level="custom">
               <property name="status=" value="PDF file transferred"/>
            </log>
            <drop/>
         </inSequence>
      </target>
      <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
      <parameter name="transport.PollInterval">15</parameter>
      <parameter name="transport.vfs.MoveAfterProcess">file:///Users/evanthika/Downloads/vfs/out</parameter>
      <parameter name="transport.vfs.FileURI">file:///Users/evanthika/Downloads/vfs/in</parameter>
      <parameter name="transport.vfs.MoveAfterFailure">file:///Users/evanthika/Downloads/vfs/failure</parameter>
      <parameter name="transport.vfs.FileNamePattern">.*\.pdf</parameter>
      <parameter name="transport.vfs.ContentType">application/pdf</parameter>
      <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   </proxy>


Now drop the relevant PDF file to the location mentioned in the transport.vfs.FileURI
parameter. After the time specified in the transport.PollInterval parameter, the relevant PDF file will be read and moved to the folder specified as the transport.vfs.MoveAfterProcess parameter value.

Saturday, September 2, 2017

How to accept requests from different URLs that has different query parameters through ESB APIs

Products used - WSO2 ESB 4.8.1
                          WSO2 DSS 3.5.0


Assume that we have a back-end service which reads a database and returns employee information depending on particular parameters that are being passed. Let's say that we are using the following Data Services Server which is hosted in WSO2 DSS product.

<data name="GetEmployees" transports="http https local">
   <config enableOData="false" id="mysql">
      <property name="driverClassName">com.mysql.jdbc.Driver</property>
      <property name="url">jdbc:mysql://localhost:3306/employee</property>
      <property name="username">root</property>
      <property name="password">root</property>
   </config>
   <query id="query1" useConfig="mysql">
      <sql>select * from employees where id=? and lastname=?</sql>
      <result element="employees" rowName="employee">
         <element column="id" name="id" xsdType="string"/>
         <element column="lastname" name="lastname" xsdType="string"/>
         <element column="firstname" name="firstname" xsdType="string"/>
      </result>
      <param name="id" sqlType="STRING"/>
      <param name="lastname" sqlType="STRING"/>
   </query>
   <query id="query2" useConfig="mysql">
      <sql>select * from employees</sql>
      <result element="employees" rowName="employee">
         <element column="id" name="id" xsdType="string"/>
         <element column="lastname" name="lastname" xsdType="string"/>
         <element column="firstname" name="firstname" xsdType="string"/>
      </result>
   </query>
   <query id="query3" useConfig="mysql">
      <sql>select * from employees where id=?</sql>
      <result element="employees" rowName="employee">
         <element column="id" name="id" xsdType="string"/>
         <element column="lastname" name="lastname" xsdType="string"/>
         <element column="firstname" name="firstname" xsdType="string"/>
      </result>
      <param name="param0" sqlType="STRING"/>
   </query>
   <operation name="getemployee">
      <call-query href="query1">
         <with-param name="id" query-param="id"/>
         <with-param name="lastname" query-param="lastname"/>
      </call-query>
   </operation>
   <operation name="getemployeeid">
      <call-query href="query3">
         <with-param name="param0" query-param="param0"/>
      </call-query>
   </operation>
   <operation name="getAllEmployees">
      <call-query href="query2"/>
   </operation>
</data>


Lets assume that the client expects to send the requests in the following format.

To get all the employee details of the database - http://localhost:8280/newsample/employee/get/employees

To get employee details which matches a particular id - http://localhost:8280/newsample/employee/get/employees?id={id_number}

To get details of a particular employee which matches a particular id and the lastname - http://localhost:8280/newsample/employee/get/employees?id=1&lastname=Amarasiri

To support this, we can create an API in WSO2 ESB with the following configuration.

      <api name="EmployeeDetApi" context="/newsample">
      <resource methods="GET"
                uri-template="/employee/get/employees?id={id}&lastname={lastname}">
         <inSequence>
            <payloadFactory media-type="xml">
               <format>
                  <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                                    xmlns:dat="http://ws.wso2.org/dataservice">
                     <soapenv:Header/>
                     <soapenv:Body>
                        <dat:getemployee>
                           <dat:id>$1</dat:id>
                           <dat:lastname>$2</dat:lastname>
                        </dat:getemployee>
                     </soapenv:Body>
                  </soapenv:Envelope>
               </format>
               <args>
                  <arg evaluator="xml" expression="$url:id"/>
                  <arg evaluator="xml" expression="$url:lastname"/>
               </args>
            </payloadFactory>
            <property name="SOAPAction"
                      value="urn:getemployee"
                      scope="transport"
                      type="STRING"/>
            <property name="ContentType" value="text/xml" scope="axis2" type="STRING"/>
            <log>
               <property name="incoming_message"
                         value="*******GET EMPLOYEE DETAILS - id ,lastname *******"/>
            </log>
            <send>
               <endpoint key="AddressEpr"/>
            </send>
         </inSequence>
      </resource>
      <resource methods="GET" uri-template="/employee/get/employees?id={param0}">
         <inSequence>
            <payloadFactory media-type="xml">
               <format>
                  <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                                    xmlns:dat="http://ws.wso2.org/dataservice">
                     <soapenv:Header/>
                     <soapenv:Body>
                        <dat:getemployeeid>
                           <dat:param0>$1</dat:param0>
                        </dat:getemployeeid>
                     </soapenv:Body>
                  </soapenv:Envelope>
               </format>
               <args>
                  <arg evaluator="xml" expression="$url:id"/>
               </args>
            </payloadFactory>
            <property name="SOAPAction"
                      value="urn:getemployeeid"
                      scope="transport"
                      type="STRING"/>
            <property name="ContentType" value="text/xml" scope="axis2" type="STRING"/>
            <log>
               <property name="incoming_message"
                         value="*******GET EMPLOYEE DETAILS - id *******"/>
            </log>
            <send>
               <endpoint key="AddressEpr"/>
            </send>
         </inSequence>
      </resource>
      <resource methods="GET" uri-template="/employee/get/employees">
         <inSequence>
            <payloadFactory media-type="xml">
               <format>
                  <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                                    xmlns:dat="http://ws.wso2.org/dataservice">
                     <soapenv:Header/>
                     <soapenv:Body>
                        <dat:getAllEmployees/>
                     </soapenv:Body>
                  </soapenv:Envelope>
               </format>
               <args/>
            </payloadFactory>
            <property name="SOAPAction"
                      value="urn:getemployeedetails"
                      scope="transport"
                      type="STRING"/>
            <property name="ContentType" value="text/xml" scope="axis2" type="STRING"/>
            <log>
               <property name="incoming_message"
                         value="*******GET EMPLOYEE DETAILS - All employees details *******”/>
            </log>
            <send>
               <endpoint key="AddressEpr"/>
            </send>
         </inSequence>
      </resource>
   </api>

Wednesday, July 26, 2017

Analysing data with Data Analytics Server

When we talk about WSO2 DAS there are a few important things we need to give focus to. They are, Event receivers, Event Streams, Event Stream definitions and Event Stores.

Events, are units of data, that are received by WSO2 DAS using Event Receivers. Through these Event Receivers WSO2 DAS receives events from different transports in JSON, XML, WSO2 Event formats, etc. formats. There are many different Event receivers available in WSO2 DAS, such as HTTP Event Receivers, SOAP Event Receivers, WSO2Event Event Receivers, etc.

Event Streams are known to be a sequence of events of a particular type. The “type” in this context can be defined as an event stream definition.

An Event Stream definition is sort of a schema which describes in which format the events that comes into WSO2 DAS server should be in. Each Event Stream Definition would have a name, a version and most importantly the type of the data that it expects to be sent in to WSO2 DAS as Events.

Once an event is received by the Event Receiver, it would be checked against the Event Stream definition and be persisted to an Event Store. This is happening through the Data Analytics Layer (DAL) where the events will be stored in the Event store (can be a relational database) as a BLOB which is in human unreadable format. Then these events will be analyzed and processed and the processed information will then again be store in a Process Store (This too can be a relational database) in a BLOB format.
These analyzed data will be decrypted by DAL and presented in a human readable format through the Data Explorer of WSO2 DAS.

When it comes to IS Analytics, whatever the analyzed data that are in the Process Store will be presented through the Analytics Dashboard which is available in WSO2 DAS after the data being decrypted from DAL.

However, the API Manager Analytics are visible from the Store/Publisher portals that are shipped with the API Manager product. However, the API manager related events that are stored in the Process store cannot be read directly from the API Manager dashboards as they are in a encrypted format. Only a DAL can decrypt this information into a human readable format. Because of this, we have introduced a way which is using a method called carbonJDBC where the DAL converts these information that are in the process store and store them in an external relational database. This database is then pointed to, from the API Manager dashboards and you will see API Manager analytics accordingly.

Tuesday, June 13, 2017

How to resolve "Un-recognized attribute 'targetFramework'. Note that attribute names are case-sensitive." in IIS

While trying to configure a SOAP service in IIS, I came across various issues where I had to do many things to resolve them. One of them is the following issue.

Server Error in '/' Application.
Configuration Error
Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately.

Parser Error Message: Un-recognized attribute 'targetFramework'. Note that attribute names are case-sensitive.

Source Error:

Line 3: 
Line 4:   
Line 5:     
Line 6:   

Line 7:   

Source File: C:\RestService\RestService2\RestService\web.config    Line: 5

Version Information: Microsoft .NET Framework Version:2.0.50727.5420; ASP.NET Version:2.0.50727.5459


To resolve this issue, what you need to do is, check whether you have installed ASP.NET 4 on your Windows instance.
If it is installed, open up a command window, go to the location where .NET 4 is available,

C:\Users\Administrator>cd C:\Windows\Microsoft.NET\Framework\v4.0.30319

Then run the following command

aspnet_regiis -i

E.g.:- C:\Windows\Microsoft.NET\Framework\v4.0.30319>aspnet_regiis -i

Once this is done, open your IIS (type inetmgr in run) and change your Application pool setting to .Net4 (Go to Application Pools -> click on your project -> Right click and select 'Basic Settings')





Sunday, June 11, 2017

Where do we have the public GIT repos for XACML and XACML Mediation Feature?

If you want to do changes to the XACML and XACML mediation features and use it within your product, you can get the code from the below locations.

 XACML
https://github.com/wso2/carbon-identity-framework/tree/master/features/xacml  

XACML Mediation Feature https://github.com/wso2/carbon-mediation/tree/master/components/mediators/entitlement/org.wso2.carbon.identity.entitlement.mediator

Monday, June 5, 2017

Some important grep commands

While working on day to day work, there are many important grep commands that comes in handy. Below are some of these commands.

  • How to find the number of occurrences of a particular text in a file.

grep -o "text to be searchd" nohup.out | wc -l
This blog is yet to be developed and I will add one by one when I come across them.

Sunday, June 4, 2017

How does thread switching happen in WSO2 ESB (Switching between ServerWorker & IO reactor threads)

One of the most important things we need to understand when working with WSO2 ESB is it's threading model. There are two types of threads that does the main functionality within the ESB.

a) SynapseWorker threads
b) IO Reactor Threads

Out of these two threads, the SynapseWorker threads handles whatever the operations related to message mediation. On the other hand, the IO reactor threads are available for event handling such as receiving message headers, receiving message body, etc.

Inside the Synapse engine, the message would go through many classes where transformations that are required are done to the incoming message. The classes involved in this flow are explained in a previous post which I have written.

Once the message mediation is done, it is passed onto the Axis2Engine where it's receive() method is called. Inside this method, the type of the receiver is read from the context. In this example, since we are sending the message to a Proxy service, the receiver would be set to ProxyServiceMessageReceiver. So the message context would be passed on to the receive method of the ProxyServiceMessageReceiver class.
This is the point where the incoming message is passed onto the Synapse Engine.



After the mediation flow completes for the incoming message inside the mediation engine, it would be handed over to the PassThroughHttpSender where the outbound HTTP requests can be sent.

The PassThroughHttpSender implements the TransportSender interface. The initialisation of the PassThroughHttpSender happens at the server startup and an instance of the NHttpClientEventHandler is created by the name TargetHandler.

In addition to this, when the PassThroughHttpSender is initialised, it creates an instance of the DefaultConnectingIOReactor as well.
Along with this an instance of the DeliveryAgent is also formed which allows storing of messages for later delivery.
If a connection is available to send the message it would send the message right away and if not, it would put the message to a message queue to be delivered to the backend whenever a connection is available. This implementation is available inside the submit() method of the DeliveryAgent class.

When the message comes in the PassThroughHttpSender it hits it's invoke() method. After removing unwanted headers sent through the request, it would then check the messageContext for the endpoint value sent in the request message.
If an endpoint is passed in from the request, the submit() method of the DeliveryAgent class is called. If no endpoint is sent, it would call the submitResponse() method.

In this scenario, since we have specified an endpoint value, I would explain the flow which continues from the submit() method of the DeliveryAgent class.



Inside the submit() method, it would add the incoming message to a queue as shown below.



Once the message is added to the queue, it would call the getConnection() method of the TargetConnections class. This would return a connection if already available or won't return anything and will notify the delivery agent when a connection is available.



In a scenario where a connection is available, the connected() method of the TargetHandler is called and also a new TargetContext is created and the Protocol state is set to REQUEST_READY. which means that the connection is at the initial stage and is ready to receive a request.


Next the connected() method of the DeliveryAgent class is invoked where it would check the queue for any messages and pass onto the tryNextMessage() method.



Inside this method, the TargetConext will be updated with the status REQUEST_READY. At the same time, the TargetContext will be set with the messageContext and then the message will be sent to the submitRequest() method.



When the submitRequest() method is called, it would create a TargetRequest and attach it to the TargetContext.


Next is the invocation of the requestReady() method of the TargetHandler class where the HTTP headers are written to the TargetRequest.




Then the outputReady() of the TargetHandler is called;  where the write() method of the TargetRequest is hit.



In this method, it reads the data that was written before to the pipe and writes it to the wire. Once this is done, the Protocol status is also updated to REQUEST_DONE.



There you go! Now you know how the request messages are being passed on from the Worker threads to the IO threads within WSO2 ESB.