Commit Graph

338 Commits

Author SHA1 Message Date
Michael Gmelin
43d6a8ab09 MFC r297052:
Update fetch.1 and fetch.3 to reflect libfetch's actual use of CA bundles
2016-06-06 11:08:05 +00:00
Don Lewis
44ffc0c54a MFC r300666
Call closedir() before returning from fetchListFile() to avoid a leak.

Reported by:	Coverity
CID:		1016697
2016-06-01 17:45:00 +00:00
Don Lewis
dee32aa9c8 MFC r300665
Don't leak addrinfo in fetch_bind()

Submitted by:	Coverity
CID:		1225038
2016-06-01 17:43:04 +00:00
Dag-Erling Smørgrav
dec1d4d8ec MFH (r294326): fall back to standard / configured CA store
PR:		193871
Approved by:	re (gjb)
2016-02-20 22:58:33 +00:00
Dag-Erling Smørgrav
de9066ff28 MFH (r295536): fix double-free error when SSL connection fails
PR:		206774
Submitted by:	Christian Heckendorf <heckendorfc@gmail.com>
Approved by:	re (glebius)
2016-02-17 11:40:03 +00:00
Dag-Erling Smørgrav
e540cd5a1e MFH (r280630): remove all traces of SSLv2 support
MFH (r285141): remove unused variable
MFH (r288217): correctly check return value from getaddrinfo(3)
MFH (r289419): fix bugs in HTTPS tunnelling
MFH (r289420): use fopen()'s "e" mode instead of fcntl for close-on-exec
MFH (r291453, r291461): use .netrc for http servers and proxies
MFH (r292330, r292332): reset bufpos to 0 after refilling in chunked mode

PR:		194483 199801 193740 204771
2016-01-16 20:24:02 +00:00
Dimitry Andric
bdf150f7b1 MFC r284346:
Fix the following clang 3.7.0 warnings in lib/libfetch/http.c:

    lib/libfetch/http.c:1628:26: error: address of array 'purl->user'
    will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
                                    aparams.user = purl->user ?
                                                   ~~~~~~^~~~ ~
    lib/libfetch/http.c:1630:30: error: address of array 'purl->pwd'
    will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
                                    aparams.password = purl->pwd?
                                                       ~~~~~~^~~~
    lib/libfetch/http.c:1657:25: error: address of array 'url->user'
    will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
                                    aparams.user = url->user ?
                                                   ~~~~~^~~~ ~
    lib/libfetch/http.c:1659:29: error: address of array 'url->pwd'
    will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
                                    aparams.password = url->pwd ?
                                                       ~~~~~^~~ ~
    lib/libfetch/http.c:1669:25: error: address of array 'url->user'
    will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
                                    aparams.user = url->user ?
                                                   ~~~~~^~~~ ~
    lib/libfetch/http.c:1671:29: error: address of array 'url->pwd'
    will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
                                    aparams.password = url->pwd ?
                                                       ~~~~~^~~ ~

Since url->user and url->pwd are arrays, they can never be NULL, so the
checks can be removed.

Reviewed by:	bapt
Differential Revision: https://reviews.freebsd.org/D2673
2015-06-20 13:30:09 +00:00
Dag-Erling Smørgrav
d15c8ff78c MFH (r273114, r273124): disable SSLv3 by default. 2015-01-12 10:02:23 +00:00
Dag-Erling Smørgrav
babfe5dc4a MFH (r267127): don't send User-Agent if HTTP_USER_AGENT is empty 2014-08-24 14:04:20 +00:00
Baptiste Daroussin
525e7e22a1 MFC: r267131, r267132, r267133, r268493, r268671
Use NULL instead of 0 (Patch by Sascha Wildner <saw at online.de> for Dragonfly)
Remove unnecessary semicolons (Patch by Sascha Wildner <saw at online.de> for Dragonfly)
Add support for arbitrary http requests [1]
Support EAGAIN in fetch_writev

Submitted by:	Alex Hornung <alex at alexhornung.com> [1]
Reviewed by:	des
2014-07-20 00:29:41 +00:00
Dag-Erling Smørgrav
a8941797ca MFH (r266291): look for root certs in /usr/local first 2014-05-24 22:50:58 +00:00
Bryan Drewery
3186b95e0a MFC r263021:
Support Last-Modified behind proxies which return UTC instead of GMT.
2014-03-19 00:53:24 +00:00
Dag-Erling Smørgrav
b314edf70e MFH (r260904): fix format string
MFH (r261230, r261263): fix buffering issues
MFH (r261284): bump copyright
2014-02-27 13:27:04 +00:00
Bryan Drewery
4b49d828fd MFC r258347,r258349:
Support SNI in libfetch

  SNI is Server Name Indentification which is a protocol for TLS that
  indicates the host that is being connected to at the start of the
  handshake. It allows to use Virtual Hosts on HTTPS.

PR:		kern/183583
Approved by:	bapt (implicit)
Approved by:	re (gjb)
2013-11-29 21:13:30 +00:00
Dag-Erling Smørgrav
615c5740ef Even though it doesn't really make sense in the context of a CONNECT
request, RFC 2616 14.23 mandates the presence of the Host: header in
all HTTP 1.1 requests.

PR:		kern/181445
Submitted by:	Kimo <kimor79@yahoo.com>
MFC after:	3 days
2013-08-22 07:43:36 +00:00
Dag-Erling Smørgrav
1453595f49 Include an Accept header in requests.
PR:		kern/180917
MFC after:	1 week
2013-07-30 13:07:55 +00:00
Dag-Erling Smørgrav
dcd47379ff Implement certificate verification, and many other SSL-related
imrovements; complete details in the PR.

PR:		kern/175514
Submitted by:	Michael Gmelin <freebsd@grem.de>
MFC after:	1 week
2013-07-26 15:53:43 +00:00
Dag-Erling Smørgrav
ba7c6aec97 Use the correct request syntax for proxied (tunneled) HTTPS requests.
PR:		bin/180666
MFC after:	3 days
2013-07-21 06:59:56 +00:00
Tim Kientzle
9bc22394d8 Fix -Wunsequenced warning.
Submitted by:	dt71@gmx.com
2013-06-29 15:51:27 +00:00
Dag-Erling Smørgrav
4056bae982 Use the CONNECT method to proxy HTTPS connections through HTTP proxies.
PR:		bin/80176
Submitted by:	Yuichiro NAITO <naito.yuichiro@gmail.com>
2013-04-12 22:05:15 +00:00
Dag-Erling Smørgrav
eab7a548ba Fix weird indentation. 2012-11-16 12:31:43 +00:00
Eitan Adler
8d049fb235 Implement HTTP 305 redirect handling.
PR:		172452
Submitted by:	gcooper
Reviewed by:	des
Approved by:	cperciva
MFC after:	1 week
2012-10-22 03:00:15 +00:00
Eitan Adler
c4fa1489ec Don't deny non-temporary redirects if the -A option is set (per
the man page) [0]

While here add support for draft-reschke-http-status-308-07

PR:		172451 [0]
Submitted by:	gcooper [0]
Reviewed by:	des
Approved by:	cperciva
MFC after:	1 week
2012-10-22 03:00:10 +00:00
Eitan Adler
e6c0e200f4 Be a bit more lenient in the maximum number of redirects allowed.
Chrome and Firefox have a limit of 20. IE has a limit of 8.

Reviewed by:	des
Approved by:	cperciva
MFC after:	3 days
2012-10-22 03:00:04 +00:00
Dag-Erling Smørgrav
0e50a83330 Use libmd if and only if OpenSSL is not available.
PR:		bin/171402
MFC after:	3 days
2012-09-14 13:00:43 +00:00
Eitan Adler
42415fcb52 Adding missing return statements during error conditions.
PR:		kern/171187
Submitted by:	Mark Johnston <markjdb@gmail.com>
Reviewed by:	des
Approved by:	cperciva
MFC after:	2 weeks
2012-09-14 12:15:13 +00:00
Jung-uk Kim
1f13597d10 Merge OpenSSL 1.0.1c.
Approved by:	benl (maintainer)
2012-07-12 19:30:53 +00:00
Jilles Tjoelker
28fd93073a libfetch: Avoid SIGPIPE on network connections.
To avoid unexpected process termination from SIGPIPE when writing to a
closed network connection, enable SO_NOSIGPIPE on all network connections.

The POSIX standard MSG_NOSIGNAL is not used since it requires modifying all
send calls to add this flag. This is particularly nasty for SSL connections.

Reviewed by:	des
Tested by:	bapt
MFC after:	5 days
2012-05-28 19:22:23 +00:00
Dag-Erling Smørgrav
f51b84bcc4 Don't reuse credentials if redirected to a different host.
Submitted by:	Niels Heinen <heinenn@google.com>
MFC after:	3 weeks
2012-04-30 12:12:48 +00:00
Dag-Erling Smørgrav
30787285b5 Since the socket is non-blocking, it is necessary to use select(2) even
when there is no timeout, because read(2) will return immediately if there
is no data waiting in the TCP buffer, causing fetch_read() to busy-loop on
slow connections.

MFC after:	3 weeks
Noticed by:	Yanhui Shen <shen.elf@gmail.com>
2012-04-30 12:11:45 +00:00
Ed Maste
0fa391997a Support percent-encoded user and password
RFC 1738 specifies that any ":", "@", or "/" within a user name or
password in a URL is percent-encoded, to avoid ambiguity with the use
of those characters as URL component separators.

Reviewed by:    rstone@
MFC after:	1 month
2012-04-11 15:42:02 +00:00
Eitan Adler
50d675f7a9 Remove trailing whitespace per mdoc lint warning
Disussed with:	gavin
No objection from:	doc
Approved by:	joel
MFC after:	3 days
2012-03-29 05:02:12 +00:00
Dag-Erling Smørgrav
e6e3bcd376 Fix two nits in previous commit pointed out by pjd@.
MFC after:	3 weeks
2012-01-23 09:23:07 +00:00
Dag-Erling Smørgrav
2a7daafe67 Fix two issues related to the use of SIGINFO in fetch(1) to display
progress information.  The first is that fetch_read() (used in the HTTP
code but not the FTP code) can enter an infinite loop if it has previously
been interrupted by a signal.  The second is that when it is interrupted,
fetch_read() will discard any data it may have read up to that point.
Luckily, both bugs are extremely timing-sensitive and therefore difficult
to trigger.

PR:		bin/153240
Submitted by:	Mark <markjdb@gmail.com>
MFC after:	3 weeks
2012-01-18 15:13:21 +00:00
Dag-Erling Smørgrav
578153f1ba latin1 -> utf8 2011-10-19 11:43:51 +00:00
Dag-Erling Smørgrav
6337341d81 Update copyright dates and strip my middle name. 2011-09-27 18:57:26 +00:00
Dag-Erling Smørgrav
76b94eb6e9 Bump date. 2011-09-27 18:53:35 +00:00
Dag-Erling Smørgrav
ecd18c961b Think first, commit second.
1. Allow the caller to select active mode.
2. Fix the envar logic so it *always* overrides the caller's flags.
3. Document the change from active to passive.
2011-09-27 18:42:09 +00:00
Dag-Erling Smørgrav
4f411f8601 Long overdue: make passive mode the default for ftp. 2011-09-27 18:23:58 +00:00
Dag-Erling Smørgrav
15b68c63bb Mark all socket and file descriptors close-on-exec.
PR:		bin/151866
MFC after:	3 weeks
2011-05-13 07:21:41 +00:00
Dag-Erling Smørgrav
ee3cb0e502 Builds cleanly at the default WARNS level (WARNS=6). 2011-05-12 21:30:46 +00:00
Dag-Erling Smørgrav
eb9b80c30d Increase WARNS to 4. 2011-05-12 21:26:42 +00:00
Dag-Erling Smørgrav
c12c6e3cda Mechanical whitespace cleanup. 2011-05-12 21:18:55 +00:00
Dag-Erling Smørgrav
a42eecded0 Increase WARNS to 3. 2011-05-12 21:12:24 +00:00
Dag-Erling Smørgrav
e8eaf30078 Fix a bug related to connection caching which could cause a crash after
a STAT command.

PR:		kern/153748 (different patch)
Submitted by:	Mark Johnston <markjdb@gmail.com>
MFC after:	2 weeks
2011-01-17 17:25:46 +00:00
Ed Maste
a9d0c84909 Move variable declarations into the conditional block where they are
used, to fix warning if WITH_SSL is not set.

Submitted by:	Sean Bruno
MFC after:	1 week
2010-10-24 01:05:10 +00:00
Dag-Erling Smørgrav
caaffed8f0 Redo fetch_read() using non-blocking sockets. This is necessary to
avoid a hang in the SSL case if the server sends a close notification
before we are done reading.  In the non-SSL case, it can provide a
minor (but probably not noticeable) performance improvement for small
transfers.

MFC after:	3 weeks
2010-07-28 16:11:22 +00:00
Dag-Erling Smørgrav
c954ded250 Fix a couple of embarrassing mistakes in the previous commit.
Submitted by:	Dimitry Andric <dimitry@andric.com>
2010-07-28 15:29:18 +00:00
Dag-Erling Smørgrav
962cf29525 If the A flag is supplied, http_request() will attempt the request only
once, even if authentication is required, instead of retrying with the
proper credentials.  Fix this by bumping the countdown if the origin or
proxy server requests authentication so that the initial unauthenticated
request does not count as an attempt.

PR:		148087
Submitted by:	Tom Evans <tevans.uk@googlemail.com>
MFC after:	2 weeks
2010-07-01 17:44:33 +00:00
Dag-Erling Smørgrav
78122aac3d Document HTTP digest authentication support.
Mention some prominent past contributors: Hajimu Umemoto (ipv6), Henry
Whincup (https), Jukka Ukkonen (if-modified-since) and Jean-François
Dockes (digest auth)
2010-01-26 15:06:22 +00:00