The business of ftp

FTP gist
- 21 command port
- data transfer port: server determined

- SFTP - ftp over ssh
- FTPS - ftp over TLS

About FTP:
https://winscp.net/eng/docs/ftp_modes
About FTP: PORT and PASV
https://community.ipswitch.com/s/article/An-explanation-of-PORT-and-PASV-connections-1307565978853
About how the port command works,
especially how to translate server message into ip and port to bind to for data
https://searchnetworking.techtarget.com/tip/Understanding-the-FTP-PORT-command


Most of the time servers are misconfigured.
So, MO:
- explore the idiosyncrasies of the server using lftp
-  most likely the ftp lib of the language would need customisation to handle the non-standard implementations

Best tools to examine a server and its configs:
- lftp
http://manpages.ubuntu.com/manpages/xenial/man1/lftp.1.html


Run scripts with:
lftp -f scripts/lftp_script.lftp

Eg. script:
-x-x-x-
debug 10 # debug is not very well documented
set ssl:verify-certificate false
set ftp:ssl-force true
set ftp:ssl-protect-data true
open ftp://username:password@localhost:6021

ls

-x-x-x-

https://serverfault.com/a/59914

- lftp can even mirror entire servers

- all lftp features are designed to start from where they left off

- 

---------------------------------------------------------------------

Curl can also be used for exploratory work (with the --verbose flag)

- never tried it personally.

- lftp has not fallen short so far

---------------------------------------------------------------------

Telnet can also be used for very low level exploration if required:
https://www.ntchosting.com/encyclopedia/ftp/ftp-port-connection/


Eg.
telnet my-best-domain.net 21

Trying 192.128.34.174...

Connected to my-best-domain.net.

220 ProFTPD 1.2.10 Server (my-best-domain.net) [192.128.34.174]     

---------------------------------------------------------------------

Tunnelling FTP / FTPS:
- 2 tunnels required
- one for cmd on a fixed port (generally on port 21)
- second for data response (if you are lucky on a limited range of ports)

if the second port is not following a pattern:
- use socks proxy (dynamic port forwarding in ssh)

https://albert.cx/20190221-use-ftp-lftp-over-socks-proxy

Use FTP or LFTP over a SOCKS proxy or SSH tunnel

Install tsocks:

sudo dnf install tsocks  # OR ...
sudo apt install tsocks

Start a SOCKS tunnel over SSH:

ssh -D 6000 my.tunnel.host

Create a ~/.tsocks.conf file:

cat > ~/.tsocks.conf <<EOF
server = 127.0.0.1
server_port = 6000
EOF

Run your FTP client using tsocks and the newly created config file:

TSOCKS_CONF_FILE=$HOME/.tsocks.conf tsocks lftp my.ftp.host

---------------------------------------------------------------------

Python and ftp:
- https://stackoverflow.com/questions/18772703/read-a-file-in-buffer-from-ftp-python
- https://docs.python.org/3/library/ftplib.html#ftplib.FTP


Server idiosyncrasies:

https://access.redhat.com/solutions/336113

• Why do I get "534 Protection level negotiation failed." error when attempting to transfer files to a Microsoft Internet Information Server FTPS
  service using lftp?
  
• The IIS FTPS service is configured with "Require SSL connections".
  

Well this is a genuine one.
But yeah, Microsoft servers are shitty in general.