'Azure Data Factory route to external SFTP without SHIR

ADF with AIR connected to external SFTP server to grab file. This SFTP have firewall that have to whitelist IP address where connection come from. Trafix should be routed via Managed NET private endpoint and then some how to NAT gateway and public IP.

Is it possible to implement such thing ?

We want to avoid using any IaaS/VMs.

azureazure-data-factoryazure-virtual-network


Solution 1:[1]

• Yes, you surely can implement such thing in your Azure environment considering that the SFTP server from which you have to grab a file is also hosted in Azure protected by an Azure firewall or even if it is not and if the SFTP server is hosted in on-premises protected by a hardware firewall, then also the setup for implementing this is quite possible.

For the above purpose, you will have to deploy a private link service and a load balancer with public IP address enabled in your tenant. Also, ensure that the load balancer is hosted in a virtual network subnet created in the same region and resource group where Azure Data Factory with Azure Integrated runtime is deployed. Since you have already deployed the ADF with AIR in the managed virtual network with private endpoints, it is possible for this managed private endpoint in an approved state to send traffic to a given private link resource.

• Once you create a private endpoint connection, it is in a ‘Pending’ state unless approved by the resource owner. If the owner approves the connection, a private link is established. Otherwise, the private link won't be established. In either case, the managed private endpoint will be updated with the status of the connection. Once, the private endpoint is setup, then you can setup the load balancer with the SFTP server’s public IP address as the backend and the private endpoint of the ADF with AIR as the frontend for you to fetch the file from the SFTP server and initiate the connection from the ADF. Kindly refer to the below reference diagram and documentation link given for more information in this regard as to how to implement the said setup using the above stated Azure resources.

ADF Architecture

https://docs.microsoft.com/en-us/azure/data-factory/tutorial-managed-virtual-network-on-premise-sql-server

In the above link, you have to connect to an on-premises SQL server instead of a SFTP server and the ADF managed private endpoint is shown connecting to another allowed Azure service resources for which instead, in your case, you can directly connect it to the Private Link service deployed in as stated earlier.

Solution 2:[2]

I found two options to do this:

  1. Run SHIR at VMSS with custom extention installation.
  2. Run SHIR at Windows Container at AKS. However AKS use VMSS behind, so VMSS looks more simple for some cases.

My goal was avoid using IaaS services to connect to external (non-Azure) SFTP server.

possible options

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 KartikBhiwapurkar-MT
Solution 2 Alezis

Related Questions